Anonymization Gateway for Datagrid VCTR
The examples below set up the gateway in its default configuration, which can be used immediately to access the Datagrid API and to send telemetry data safely without disclosing any host identifying information. Simply configure the Datagrid clients to use your gateway host instead of the default
The gateway is available as a Docker application container, which can be run on any system that has Docker (see References for info on installing Docker)
Start with new key and auto-generated self-signed certificate:
sudo docker pull datagridsys/agate sudo mkdir /etc/dgri sudo touch /etc/dgri/agate.conf sudo docker run -d --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \ -p 8080-8081:8080-8081/tcp --name agate --hostname agate datagridsys/agate \ server --autokey mygateway.example.com
Note: in the above example, replace
mygateway.example.comwith hostname or IP address for the gateway
RPM from Datagrid Yum Repo (CentOS-7, RHEL-7)
Configure access to the Datagrid yum repository and install with yum:
sudo rpm -U https://s3.amazonaws.com/dgri-bits/download/dgri-rpm-repo-latest.noarch.rpm sudo yum install -y epel-release sudo yum install -y dgri-gateway
First run - start with new key and auto-generated self-signed certificate.
sudo dgri-gateway server --autokey mygateway.example.com
NOTE: this runs the server in the foreground. You can stop it with Ctrl-C and run it as a service with 'sudo systemctl start dgri-gateway'. It will also start automatically on system boot.
(NOTE: tar file not published yet, ignore this section unless you received the dgri-gateway tar package from Datagrid directly)
Use this option to run on OS-es that are not supported by a pre-made package or if you want to download and run the gateway without having root/administrator privilege. Note that while you do not need to be root to run the gateway, installing some of the prerequisites may require this.
curl https://s3.amazonaws.com/files.datargidsys.com/dgri-gateway-1.0.0a4.tar.gz | tar -zxv cd dgri-gateway-1.0.0a4 mkdir -p ~/.dgri touch ~/.dgri/agate.conf
sudo apt-get install openssl,
sudo yum install openssl, or similar )
sudo apt-get install python-crypto,
sudo yum install python2-crypto, ... )
sudo apt-get install python-webpy,
sudo pip install web.py; note web.py is not available as RPM)
sudo apt-get install python-openssl,
sudo pip install pyOpenSSL; or
yum install python-opensslafter setting up the Datagrid repository - see RPM install above.)
You can now run the
dgri-gateway executable from the current directory as
./dgri-gateway. Startin with a new key and auto-generated self-signed certificate:
./dgri-gateway server --autokey mygateway.example.com
dgri-gateway [--nossl] (version | genkey | getkeys | getconfig | setconfig | gencert) dgri-gateway server [--nossl][ --autokey fqdn.hostname ]
All commands except
server are transient and exit immediately upon completion.
Displays program version and exits
Generates a new encryption key and adds it to the configured keys. Up to three keys are kept. If there are already 3 keys stored, the oldest one is deleted.
If there is a running server, it needs to be reloaded or restarted for the new key to take effect. See Signals.
Prints the current configuration to stdout. This command is intended for configuration backup and to replicate the same configuration to multiple instances of the gateway. See
setconfig for a special note about the server certificate, if using multiple concurrently running copies of the gateway.
getconfig command output can be given to setconfig to restore the same exact configuration.
Prints the encryption keys to stdout. Similarly to
getconfig, this command is intended for configuration backup and to replicate the same configuration to multiple instances of the gateway.
The output of
getkeys can be fed to
setconfig, it will cause only the keys to be replaced, leaving the server certificate alone. Use
setconfig when you need to synchronize the keys between two instances that have different server certificates.
reads configuration from stdin and saves it in the gateway's configuration directory (
The text given on stdin can be either one of:
- the output of a previous run of
dgri-gateway getconfig, or
- a server certificate in PEM format, with the private key included.
if using setconfig to set or update the server certificate, keep the following in mind:
- the certificate's common name (CN=...) must match the hostname that will be used by clients to connect to it (i.e., the name used in the dgri client configuration files for the
feed_urlsettings.). If using more than one instance of the gateway server, each one will likely have a different hostname, but it may still be convenient to use the same certificate for all of them. To do this, the "Certificate Subject Alternative Name" extension can be used, to define multiple DNS names that are matched by the same certificate. Alternatively, if the hostnames allow for this, a wildcard certificate can be used, e.g.,
- the certificate must include the private key in unencrypted form. The private key must be first in the file, followed by the certificate. If you use
openssl req -x509to produce a self-signed certificate and give it the same file name for both the -keyout and the -out options, it will write the file in the correct order (key first, then certificate).
Starts the server. The command will run in foreground, with error messages printed to 'stderr'. This is intended primarily use in a Linux container, where the containerized process is expected to stay in the foreground and not fork(). System task/session managers such as Upstart or Systemd also provide convenient means for running a foreground process as a daemon (and will also provide logging of 'stderr' into file).
Before starting the server, the configuration directory must be populated with all required files, unless the
--autokey option is used. It is also possible to start the server without SSL, using
--nossl option. The latter is not recommended, except in cases when the gateway is running in a protected network in which all hosts are trusted.
--autokey option can be used for a quick setup of a new gateway server. If the configuration directory does not contain any keys, it will generate a new key and a self-signed server certificate and then start the server immediately. If there are keys in the configuration directory,
--autokey has no effect and the server will start with the existing configuration. This behavior is useful when
dgri-gateway is ran in a container, because the command line for the container is set on first run and cannot be changed.
--autokey is specified on the command line, it must be followed by a hostname. The hostname is used as the subject of the generated certificate. The hostname should be the exact name under which the server host is known to the clients that will access the gateway. This hostname will typically be a fully-qualified DNS domain name, but this is not a requirement. If your environment uses host resolution via /etc/hosts, use whatever name refers to the server in the /etc/hosts files of the clients that will access it.
Creates a new self-signed certificate and saves it for use on the next start/re-start. If a certificate was already installed with a previous
start --auto-key or
setconfig, it will be renamed (with a warning message showing the name of the file) and the new certificate will be written in its place.
Configuring Local Authentication
By default, the gateway does not do any request authentication and forwards all incoming requests, expecting them to be authenticated by the back-end server. This requires that all hosts that connect to it (both for telemetry data reporting and for API access) to be configured with your Datagrid account credentials. Such a setup is suitable for the cases where the gateway is used as a single point of outgoing connections to the Datagrid servers, for an otherwise closed network.
If the network is open or it is otherwise undesirable to configure all reporting hosts with the Datagrid login, the gateway can be set up to hold the Datagrid credentials and be the only host that does so. Hosts that connect to the gateway are then configured to use a different user ID and pasword, which are verified at the gateway.
Enabling Local Authentication
Create a temporary text file containing the following settings:
[auth] handler = basic_local api_username = username for API input feed_username = username for Feed input api_password = password for API input feed_password = password for Feed input api_remote_username = user ID for API output feed_remote_username = user ID for Feed output api_remote_password = password for API output feed_remote_password = pass for Feed output
Note the file should be safeguarded, given that it contains login information (both for accessing the gateway and for your Datagrid account). Edit it on a trusted host and if needed set the file permissions to your own user only.
setconfig command with this file as input, e.g.:
cat /tmp/authfile | dgri-gateway setconfig
The temporary file can now be deleted.
If the gateway daemon was running, it needs to be restarted for the configuration to take effect.
It is also possible to use
setconfig to edit part of the configuration (e.g., to change only the local passwords):
printf '[auth]\n api_password = newpassword\n' | dgri-gateway setconfig
If the daemon is used as a standalone executable and you have direct access to its configuration directory (see Configuration below), the auth. configuration file can be created or edited manually, as well. It is located in
~/.dgri/agate-auth.conf. NOTE the file does not have the cleartext local passwords (
feed_password). If you edit it directly, you will find
feed_hash, instead. The hashes are in the form used by the GNU
crypt() function (in SHA-512 mode) and can be replaced manually, if you know how to use the
crypt utility or the
crypt() library function to hash your passwords.
Configuration (standalone usage)
The following information is only for using the gateway as a standalone executable (ignore this section if using the gateway as a Docker container; in that case all configuration is done only with the
The configuration is stored either in
/etc/dgri or in
~/.dgri (checked in this order. The first directory containing
agate.conf is assumed to be the configuration directory and all other files are read from it). Only one of these directories should be created on the host where you run the gateway, to avoid mistakes. Note that some of the configuration is done by running the gateway executable itself and it needs to have the directory writable. In addition, when running the server, it will require that the sensitive files are readable only to itself, therefore
/etc/dgri can be used only if running as 'root'.
dgri-gateway for the first time, create the configuration directory and an empty config file, e.g.:
mkdir ~/.dgri touch ~/.dgri/agate.conf
Create a random encryption key using this command:
Create a self-signed server certificate (or have a certificate signing request sent to a CA and use the certificate signed by your CA) and store it in the configuration directory as
agate-cert.pem. The certificate must be in PEM format and contain the private key (not locked with a password).
agate.conf Configuration Options
The configuration settings are defined in
agate.conf in the form
key = value. Note that no quotes should be placed around value. Spaces before or after the = are allowed but not required.
Example (the built-in defaults):
api_listen = *:8080 feed_listen = *:8081
The following settings are defined:
||hostname/address and port to listen to for API requests. This can be specified either as host or as host:port. If port is not set, 8080 is used. if host is
||hostname/address and port to listen to for telemetry sending requests. The format is the same as for
Handling the Encryption Keys
The encryption keys are stored in the configuration directory under variable names, in the form
agate.??.key. These files should not be edited manually, nor renamed or copied by any means other than one of:
- using the
- saving and restoring the key files (or the entire config directory) with
rsync(with the archive option
-a) or another command that preserves all file attributes.
New keys can be added with the
The gateway will use the latest key added with
genkey to encrypt outgoing data. When retrieving encrypted data and decrypting it, it will use the key that was used to encrypt it (if it is still available), allowing for up to 2 'old' keys to be used for older data that was encrypted before the
genkey was ran.
If using multiple concurrently running copies of the gateway that serve the same account, the key sets should be kept in synch between them with minimal periods of time where one copy might be running with keys that are different from the others, to avoid the possibility that decryption of retrieved data fails.
The following sequence can be used to do a near-simultaneous update of keys at several locations:
Make a new key at one of the locations, but do not reload the daemon. Then get the resulting new set of keys with
dgri-gateway genkey dgri-gateway getkeys >/tmp/newkeys
Copy the file to all other locations, update the keysets, but do not reload the daemon:
dgri-gateway setconfig </tmp/newkeys
Reload all instances of the daemon (best done with a tool that can run the same command on multiple remote locations via ssh):
killall -HUP dgri-gateway
Usage Examples - Package Install or Standalone Executable
In the examples below, the current user is assumed to be 'root' and the dgri-gateway is assumed to be accessible in the shell path (this will be the case if it was installed from a package).
If you have just downloaded the
tar file and unpacked it as a regular user (not 'root'), you can adapt the examples below by running the executable as
./dgri-gateway and replacing
~/.dgri (note you should not have an /etc/dgri directory if running as non-root).
Generate key if needed and run in the background
# default config (empty file is OK), if running as root: mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf # start in background dgri-gateway server --autokey </dev/null >logfile 2>&1 &
( umask 077 ; touch saved-config ; ) # make it not readable to others dgri-gateway getconfig >saved-config
Set up when you have a saved configuration
mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf dgri-gateway setconfig <saved-config dgri-gateway server </dev/null >logfile 2>&1 &
Set up with new key, when you have your own server certificate
mkdir -p /etc/dgri ; touch /etc/dgri/agate.conf dgri-gateway server genkey dgri-gateway setconfig </tmp/my-certificate.pem dgri-gateway server </dev/null >logfile 2>&1 &
Re-configuration - add a new random key
dgri-gateway genkey killall -HUP dgri-gateway
Re-configuration while running - update configuration from file
dgri-gateway setconfig <saved-config killall -HUP dgri-gateway
Re-configuration while running - replace expired certificate
dgri-gateway setconfig </tmp/my-new-certificate.pem killall -HUP dgri-gateway
Usage Examples - Docker Container
In the examples below, the current directory is assumed to contain the gateway files.
The examples use
agate for the container that runs the server. The name is arbitrary, you can choose your own.
Quick start with new key and auto-generated self-signed certificate
sudo docker run -d --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \ -p [bind-addr:]4433-4434:8080-8081/tcp --name agate --hostname agate datagridsys/agate \ server --autokey mygateway.example.com
(Note: replace 4433-4434 with the ports on which you want the gateway's input entry points to appear on your host. Replace the hostname with your host's name)
( umask 077 ; touch /tmp/saved-config ; ) # make it not readable to others sudo docker run --rm --volumes-from agate datagridsys/agate getconfig >/tmp/saved-config
First run when you have a saved configuration
# create a config directory # sudo mkdir /etc/dgri # sudo touch /etc/dgri/agate.conf # create the container, but don't run it sudo docker create --name agate --hostname agate --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri -p 8080-8081:8080-8081/tcp datagridsys/agate # apply configuration sudo docker run --rm -i -v /etc/dgri:/etc/dgri datagridsys/agate setconfig </tmp/saved-config # run server sudo docker start agate
Upgrade, keeping the configuration
sudo docker pull datagridsys/agate sudo docker create --restart=always -e https_proxy=$https_proxy -v /etc/dgri:/etc/dgri \ --name agate-new --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate:latest sudo docker stop agate sudo docker start agate-new # optional, rename or delete old: sudo docker rename agate agate-old sudo docker rename agate-new agate sudo docker rm agate
Reconfigure - add new random key
sudo docker run --rm -v /etc/dgri:/etc/dgri datagridsys/agate genkey sudo docker kill -s HUP agate
Reconfigure - set configuration from file
cat /tmp/saved-config | docker run --rm -v /etc/dgri:/etc/dgri -i datagridsys/agate setconfig sudo docker kill -s HUP agate
Usage Examples - Alternative
(Docker Use Without Host Directory Mapping)
sudo docker run -d --restart=always -e https_proxy=$https_proxy -p 8080-8081:8080-8081/tcp \ --name agate --hostname agate datagridsys/agate server --autokey mygateway.example.com
First run when you have a saved configuration
# create the container, but don't run it sudo docker create --name agate --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate # apply configuration sudo docker run --rm -i --volumes-from agate datagridsys/agate setconfig </tmp/saved-config # run server sudo docker start agate
Upgrade, keeping the configuration
sudo docker pull datagridsys/agate sudo docker create --restart=always -e https_proxy=$https_proxy --volumes-from agate \ --name agate-new --hostname agate -p 8080-8081:8080-8081/tcp datagridsys/agate:latest sudo docker stop agate sudo docker start agate-new # optional, rename or delete old: sudo docker rename agate agate-old sudo docker rename agate-new agate sudo docker rm agate
Reconfigure - add new random key
sudo docker run --rm --volumes-from agate datagridsys/agate genkey sudo docker kill -s HUP agate
Reconfigure - set configuration from file
cat /tmp/saved-config | docker run --rm --volumes-from agate -i datagridsys/agate setconfig sudo docker kill -s HUP agate
- Main https://www.docker.com/
- Install https://docs.docker.com/engine/installation/
- Quick setup on Linux OS-es:
curl -fsSL https://get.docker.com/ | sh