11 KiB
Using libuv with Erlang C-node
| date | comment |
|---|---|
| 2020-06-19 | create documentation and few tests |
| 2020-06-18 | first test with libuv and c-node |
Introduction
libuv is a multi-platform support library with a focus on asynchronous I/O. It was primarily developed for use by Node.js, but it’s also used by Luvit, Julia, pyuv, and others.
Erlang is a great piece of software, with strong isolation and many
crazy features. Unfortunately, Erlang was not designed to execute
processes in other languages and to schedule the world outside of its
own universe. Some projects are already present on net, like
erlexec,
alcove or
exile (in Elixir). Both of
these tools are good implementation and work pretty well but are not
portable. When you want to execute a process in an heterogeneous way,
you should probably use a more robust and tested library available on
the market. That why libuv is probably a great choice.
libuv offers facilities to create an event loop, support tcp and udp sockets (useless in our case), can deal with file system events, can spawn processes, manages thread pool and uses signal handling. Well, seems pretty nice.
All this article was created and tested under OpenBSD-current and with the last erlang relesease at time of writing, Erlang-R23.
Introduction to Erlang C-node
If you are using Erlang, you should probably know Distributed Erlang,
a framework integrated with OTP to create distributed systems where an
Erlang virtual machine become a single node running multiple
processes. C-node is using the same principle but instead of running
an Erlang node, the node is coded in C. The library used to create
this relation with the remote node is called libei and present with
erl_interface module. To use this library, like any other C library,
ei.h must be available.
#include <ei.h>
int
main(int argc, char *argv[]) {
return 0;
}
To compile this simple C code file, the path must be also given to the compiler.
cc -L/usr/local/lib/erlang23/usr/lib \
-I/usr/local/lib/erlang23/usr/include \
-lei -o ${source}.c ${source}
Connect to an Erlang node
A c-node need to be initialized with different kind of variables, like
the name of the c-node (a string), the name of the target node (a
string based on the domain name of the node), a cookie (a secret
string shared with the c-node and the erlang node), and the date of
the creation. Erlang offers
ei_connect_init()
function to initialize ei_cnode data structure, called ec in the
following code.
When this data-structure is initialized,
ei_connect()
function can be called. This function will try to reach the erlang
node and make a connection to it. At this moment in the execution
process, if a timer is created or an infinite loop is present, a
hidden node should be present on the erlang node. This can be checked
by executing erlang:nodes(hidden) and should return a list of the
hidden nodes connected.
Finally, the shutdown()
and close() functions can be called
when the c-node goes away. Both of these functions are present in the
standard C library.
#include <stdio.h> // fprintf()
#include <stdlib.h>
#include <ei.h> // ei_connect_init(), ei_connect()
#include <time.h> // time()
#include <sys/socket.h> // shutdown()
#include <unistd.h> // close()
int
main(int argc, char *argv[]) {
/* name of the c-node /*
char *node_name = "";
/* erlang node name */
char *target_node;
/* cookie used to connect to erlang node */
char *cookie = "";
/* time's creation of the node */
char creation = time(NULL)+1;
/* ei_cnode structure contains node's information */
ei_cnode ec;
/* connect to erlang node with previous variable */
ei_connect_init(&ec, node_name, cookie, creation);
/* open file descriptor, a reference to the connection */
int sockfd;
sockfd = ei_connect(&ec, node_target))
if (sockfd != 0)
fprintf(stderr, "ERROR: ei_connect failed\n");
/* stop the connection */
shutdown(sockfd, 1);
/* close the socket */
close(sockfd);
return 0;
}
Sending a message to an Erlang node
Where we send a message to an Erlang node.
Receiving a message from an Erlang node
Where we receive a message from an Erlang node.
Creating a working loop
Where we learn how to create a working loop, receiving and sending message to Erlang node.
Introduction to libuv
Where we learn how to create a libuv loop with a simple handler.
Create a stream
Where we learn how to create a stream.
Create a process
Where we learn how to create a process.
Create a worker pool
Where we learn how to create a worker pool.
Merging Erlang C-node and libuv (first model)
Where we merge Erlang, C-node and libuv together to make amazing things.
________________ _________________________________
| | | |
| Erlang-node | | C-node |
| _______ | | ____________ _________ |
| / \ | | | | | | |
| ( process )<---[message]--->| dispatcher |--->| manager | |
| \_______/ | | |____________| |_________| |
| | | / / / |
| ______ | | _________/ / / |
| / \ | | | |__/ / |
| ( kernel )<---[ping]--->| | workers | |__/ |
| \______/ | | |_________| | | |
| | | ||_________| | |
|________________| | | ||_________| |
|_________|__|__|_________________|
| | |
__| | |
| \_| |
| FD | \_|
|____| | \
|____| |
|____|
An Erlang node can start a c-node manually by executing manually the application and detach it like standard daemons. Or a c-node can be started by another user running in another context on the machine or on the network. The required information can be passed in different manners, by order of priority
- Configuration file (
~/.erluv.conf); - Environment variables;
- Command arguments.
The Erlang-node sends orders to the libuv C-node. These messages are received by a dispatcher, a dedicated loops for the messages, where they are converted in internal data-structure. These new messages are forwarded to the manager, aware of the running workers and, based on flags, are alterating the manager state or directly sent to the workers.
If a worker process receive messages from stdin, stdout, stderr or signals, they are received by the manager and sent to the dispatcher who convert and return them to the erlang-node.
Before doing these complexes tasks, some data-structure should be created and designed to make the communication possible between the dedicated erlang process and the c-node.
Managing states
Where we learn how to manage state between nodes.
Be prepared
What's going on when a C-node crash? Or a worker crash? Or the link
between them is dying? Well, distributed programming is about catching
messages and errors. Everything in Erlang is based on finite state
machine, where we can define different state depending of the
context. Well, liberluv and erluv should be designed in the same
way. The dispatcher is a machine converting messages, its role is to
send and receive messages. The manager get the messages and execute
orders, like creating a worker with defined arguments and environment
but, it can also kill them or forward the content of the children to
the dispatcher.
Merging Erlang C-node and libuv (second model)
__________________
| |
| erluv process |
_________________ | ____________ |
| | | | | | /
| Erlang-node |<--+---->| erluv_main | | < privsep & fork
| _______ | | | |____________| | \
| / \_ | | | _____||______ |
| ( process )\_ | | | | | | /
| \_______/ )\ | +---->| erluv_child | | < shared conn
| \_______/ ) | | |_____________| | \
| \_______/ | | ____|___ |
| | |_ | |_ ____| /
|_________________| | worker | |_ < app workers
|________| | | \
|________| |
|________|
-
Erlang-node spawn a c-node (with erlang-node user) or a c-node is manually spawned by a user on the system (e.g. privileged or unpriviliged one).
-
erluv_mainstart a connection to Erlang-node by using credentials given by the user from configuration file, environment variable or command line argument (e.g. node name, cookie...). -
erluv_mainfork and spawn a newerluv_childwith privilege separation. If start as root, application will fallback in unprivileged user (e.g. nobody). Connection to Erlang-node is shared toerluv_child. -
By default, no worker is started. c-node is waiting for instruction.
-
Erlang-node want to start a new worker and send a data-structure containing argument, environment variables and other useful information to spawn the worker.
-
The new worker is started and a process is started on the erlang-node at the same time, linking them together.
-
stdin,stdoutandstderrdefault file descriptors are shared with a stream between the Erlang-node anderluv_child/worker. -
When the worker die, a message is sent to the corresponding process and can be restarted if desired.
-
If
erluv_childcrash,erluv_mainsend a message to the node and wait an other to respawn a new child. all processes linking to the worker also die. -
If
erluv_maindoes not receive a confirmation afterDIE_TIMEOUT,erluv_maindie too.
The c-node name is based on a prefix, by default erluv_${random}.
Each c-node can share their information, like node_name, cookie,
running user but can't alter these information.
By default, environment variable on each worker are not configured.
Notes
- multi-hidden note support