Server

Overview

This project repository contains the RPC and RESTful projects. The RPC project is a server wrapper around a Simulator (TODO link) instance. Its goal is to be started on multiple machines to allow scaled-up simulations. An RPC server will try to connect to Zookeeper (TODO link). This is a service started as server inside the network. It allows nodes like the RPC servers or Software servers (aka RMIModelServer) to register themselves and be available to RESTful servers to perform simulations.

The RESTful server is an HTTP server. It listens for HTTP requests to setup and start simulations. Setup is done by specifying CarLang and SimLang files that describe the properties of vehicles in the simulation (CarLang) as when as the scenario to simulate (SimLang). On simulation start, it will ask the local Zookeeper for RPC/Software servers to perform the simulation on.

Since this project provides a distributed setup, testing and running it requires an "advanced" setup. Two ways of doing it:

• Using Docker: a series of scripts allow the building, testing and running of the setup inside docker images.
• Using manual local instances: install & start Zookeeper, install & start an RMIModelServer, build and run an RPC server then finally start a RESTful server. (Scripts are available in the respective projects to do that.)

Additionally, the Visulization project can be install, it is a Web Visualization that allows the control (and visualization) of the RESTful server.

Install

mvn clean install -DskipTests -s settings.xml

After installation, the executables can be found in restful/target and rpc/target directory. To use them, please refer to corresponding instructions: rpc and restful

Docker (optional)

• Run the docker/build_server script to build the whole project. This will also run the tests of the RPC project.
• The docker/run_test & docker/run_integration_test scripts will run all the tests of the RESTful project. (Requires seperate setup.)

The README of restful includes instructions of how to use docker to run this project.

Just have a look

If you don't want to compile the whole project and just want to have a taste of this project, you can simply run:

$ cd docker
$ docker-compose up

This will start a pure backend application. Now you can use the RESTful API to play with the simulation. API doc can be found at 127.0.0.1:8090/swagger-ui.html

To try it with visualization, download visualization project, uncomment the nginx part of docker/docker-compose.yml, comment or remove the ports field of server.

Portable version (Outdated)

A portable version for Windows of the SmartFoxServer based simulator together with a portable PostgreSQL database is available as archive. Running the portable version can be done by running the start-script.bat file, provided in the archive.

The portable version does not include the RMIModel server, thus it is also not working with external model. However, it is possible to test external models just by installing the RMIModel server along with an external model following this documentation. To enable the used model, one has to change the server configuration as explained here.

Requirements

In order the project to be built locally, the following software is required:

• Apache Maven
• Java Development Kit, Version 8+
• (optional) Git

To clone this repository, one can use the following command

      cd MontiSim
      git clone https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/simulators/server.git

or download the project as a .zip file.

NOTE: MontiSim is the directory, in which MontiSim-belonging projects should be clonned. It has to be manually created.

The RMIModelServer repository has to be cloned and compiled to allow the integration of external vehicle autopilot models (EMADL models). This can be done via the following commands:

                cd MontiSim
                git clone https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/simulators/RMIModelServer.git
                cd ./RMIModelServer
                mvn clean install -s settings.xml

The RMIModelServer is exported to the install folder. It contains a run.bat / run.sh script that starts the RMI server within the install folder and uses the autopilots folder it contains to load autopilots.

To use an external vehicle autopilot, one has to clone the corresponding repository. For the Autopilot emam vehicle model, located in Autopilot emam repository, the following commands have to be executed:

                cd MontiSim
                git clone https://git.rwth-aachen.de/monticore/EmbeddedMontiArc/simulators/EMAM-showcase.git
                
                cd ./EMAM-showcase/scripts
                main.bat

The last command main.bat will compile the model and run it in a server container to test it. One can close the server container right after its start, since the only reason the main.bat is used in current case is to generate the Autopilot dll library. This dll (os .so for linux) file can be renamed to a desired autopilot name and placed inside the autopilots folder used by the RMIModelServer. This folder contains the last version of the EMAM-showcase autopilot (AutopilotAdapter).

Installation

Configuration of Database Service

This project supports both PostgreSQL and SQLite database. Using PostgreSQL can benefit from its satisfying performance, but requires more steps to setup. On the contrast, SQLite is quite easy to setup and thus make it easier to start developing this project.

We recommend using SQLite in development and deploy the project with PostgreSQL.

Installation of PostgreSQL server

To install a PostgreSQL, the following steps have to be undertaken:

1. Download the installer for the OS of the machine, which is to run the database server
2. Install PostgreSQL server (*optionally: also pgAdmin)
3. Using the scripts provided at docs/DBscripts in server repository:
   • Create database with name mmosim using pgAdmin or psql console.
   • Create user using the commands provided in db-create-user file.
   • Import the database schema using the db-create-script (a dump of the database schema)

     • pg_dump can be used to import the database from dump file. For example:

           pg_dump -U server -h localhost mmosim < db-create-script

     • as alternative the following command can be used:

           psql -U server -h localhost mmosim < db-create-script

     • without authentication respectively:

           [pg_dump OR psql] mmosim < db-create-script

     • or use the PostgreSQL backup guide

     • NOTE: for Windows users, the pg_dump and psql commands have to be configured as PATH variables or executed as follows:

           C:/path/to/psql.exe mmosim < db-create-script

Installation of SQlite

To install and configure SQLite, follow these steps below:

1. Download SQLite from there homepage and install.
2. Create the database
   • For Mac/Linux users, use the script docs/DBScripts/sqlite-create-script.sh to create the database.
   • For windows users, run the command(this should in docs/DBScripts): sqlite3 world.db ".read sqlite-world.sql"

The generated SQLite database file will be deployed by scripts/deploy.sh or scripts/deploy.bat automatically after the project is built.

Configuration of OpenModelica

Since simulation 1.0.3, a new physical model: OpenModelica, was introduced. To use that, we need compile the models for smartfox.

For Windows users, docs/deploy.bat will get the job done.

For Linux/Mac users, the models have to be compiled by yourself. To do so, follow the instructions:

1. Download and install OpenModelica software here(Mac users are recommended to use the nightly version, other versions might be hard to install).
2. Open OMEdit that has just been installed.
3. Load the model files. These can be found here
4. Export them all as FMU files.
5. These generated .fmu files are zip files that contains a c project, we need to compile it:

   # Do the same for all *.fmu files
   
   $ unzip Chassis.fmu -d chassis
   $ cd chassis/sources && ./configure && make
   $ cd ../..
   $ zip -r Chassis.fmu chassis/

6. Copy all the .fmu files into the SFS2X/lib/ directory of smartfox.
7. Alternatively, there are pre-compiled Mac version of fmu files that you can use. See libs/fmu_for_mac/. With this you can skip step 1 to 5.

Deployment of Web application (visualization)

Check out visualization repository documentation.

Configuration

Map configuration

To change server configuration, the config.properties file located in main/resources has to be changed. Whenever the map defining the global virtual world needs to be changed, the following steps have to be undertaken:

• Change mapName property in config.properties file

• Change map property to point to the new OpenStreetMap xml file. (NOTE: root directory is the <SmartFoxServer directory>)

• Copy the new OpenStreetMap xml file into a subdirectory of <SmartFoxServer directory>. For example: <SmartFoxServer directory>/extensions/. In our case we used the following commands:

      cp <directory containing map file>/MapName.osm /opt/SmartFoxServer_2X/SFS2X/extensions/AllInOne/

  or

      copy <directory containing map file>/MapName.osm C:\Users\Administrator\SmartFoxServer_2X\SFS2X\extensions\AllInOne\

  and in config.properties the map property points to extensions/AllInOne/MapName.osm.

Whether the terrain will be flat or not is defined by flat_map property. Weather can be sunny or rainy, depending on the weather_rain property.

Sectors configuration

Sectors' dimensions are defined via map_sector_width and map_sector_length properties, i.e. size on axis X and size on axis Y. By changing map_overlap, one can define the size of the overlapping areas, shared among ajacent sectors (NOTE: in legacy logic for spawning vehicles, the overlapping areas' size had to be divisible of 3).

Height generator configuration

To define which kind of height data should be generated and used, you need to modify rwth.server.extensions.ZoneExt.java at line 249/250 and rwth.server.upload.ScenarioUpload.java at lines 156/157, 332/333 and 366/367. At each mentioned line-pair there is the possibility to pass on ZCoordinates.FROM_FILE (for using height data) and ZCoordinates.ALLZERO (for using flat data). Comment out one line and uncomment the other. Be sure to do this consistently for all mentioned lines.

Web server configuration

Room's user capacity can be set via max_sector_users property (NOTE: free version of SmartFoxServer allows maximum 100 concurrent users using the server).

Simulation configuration

Simulation frame calculation rate can be set via frame_loop_frequency property. Simulation buffer size can be set via cache_buffer property.

RMI vehicle model configuration

Setting the address and port, on which the RMIModelServer is accessible is possible via changing the rmi_host and rmi_port properties. Setting which vehicle model is to be used via the RMI client is possible via changing the v_model property.

NOTE:

• The RMI client is already implemented in the server project.
• The RMIModelServer has to be running before the server project needs to access it.
• The VehicleModel set in the configuration file has to be implemented as a Java Class with corrseponding name in the RMIModelServer, in order to be served as a remote vehicle model. Currently only the Autopilot class exists as extarnal vehicle model, therefore the v_model property can either be Autopilot or should not be listed at all, which will make the server to use the embedded basic vehicle model.

Verbose logging

To allow verbose loggin, which is important for SimulationLanguage module, one has to change the log4j.properties, located in <SmartFoxServer directory>/config (e.g. in Windows: C:\Users\Administrator\SmartFoxServer_2X\SFS2X\config). In that properties file the log4j.rootLogger property (line 25) has to be uncommented, i.e. it should look like the following:

                log4j.rootLogger=DEBUG, fileAppender, consoleAppender

Sample OSM maps

To download a map file from OpenStreetMap one can use the OSM export tool. The example map Aachen.osm can be found in the main/resources directory.

NOTE: download of very broad world is likely to fail using the GUI (Export button). Instead one can use the external sources listed in the bottom left part of the screen. We use mostly the Overpass API.

Aachen city center

• longitude: 50.7625 to 50.7869

• latitude: 6.0496 to 6.1170

                50.7869
      6.0496              6.1170
                50.7625

Aachen city and outskirts

• longitude: 50.6281 to 50.8918

• latitude: 5.8653 to 6.3281

                50.8918
      5.8653              6.3281
                50.6281

Brief concepts

Virtual world

Non-volatile data like virtual world and simulation scenarios are kept in a PostgreSQL database. The world is parsed using simulation/environment module. After parsing the map is split into regions, whose size depends on given configuration in config.properties file. Together with all map details the regions are saved in the database. Whenever a region is initialized on the server its world data is fetched from the database and loaded in WorldModel object.

Path Finding

After a map is parsed, for each sector distance matrices are calculated, which are later used by the path-finding procedure. Additionally, for each overlapping area exit/entry nodes are calculated and saved to the DB for facilitating later path-finding.

Scenarios

Scenarios together with respective vehicles/tracks are saved to the database and offered to client in a dropdown, from which one can choose which scenario to be played (visualized in the webapp). Each track have at least one track_section, which is a path contained in a single sector. All sections together define the whole path, i.e. track, the vehicle has to follow through different sectors.

Sample scenarios

One can find a Sample scenario available for download. To upload it correctly, one needs both Sample.osm and Sample.sim files.

Simulation

Each region handles a local simulation using simulation modules, managing all vehicles, pedestrians and virtual world objects currently residing in the region. Whevever a vehicle is transfered to another region, it is removed from the simulation of current region and all necessary data to proceed with vehicle's simulation is sent to the next region simulation. Result data from simulation is stored in SimulationBuffers, which are later used for visualization.

Troubleshooting

Due to incorrect initialization or usage of the distributed simulator, different errors might occur. This section focuses on several such problems.

Problems using the portable simulator

Almost all the problems that can occur whenever the portable simulator is used are related to the database.

Whenever the portable simulator is started on a machine with already installed PostgreSQL server, the portable database will NOT be used, rather the server will request the local database.

If an error occur, while the portable simulator is used, make sure the portable database is used and there are no other instances of PostgreSQL database server running on the same machine.

Scenario upload

• While uploading a scenario, an error might occur, due to incorrect file format or missing file. In such cases, on the server side a log Missing file! would be printed. Whenever a scenario is about to be uploaded, it is important that 2 files are selected, one with extension .sim and one with extension .osm. If the extension of one of the files is missing, it has to be manually added.

• While uploading a scenario, one might experience an error, caused due to incorrect syntax of the .sim file. In such cases, in the server logs one can see warning messages (only if verbose logging is enabled), or just an error message, that the scenario has not been parsed. In such case, verifying the syntax of the .sim file before uploading is the only solution.

• While uploading a scenario, the server might fail to find paths for involved vehicles. If on the server side there are many logs showing an error caused by the PathFinder, it is very likely that the path-finding preprocessing has not been initialized correctly. In such cases all the data in the database for the used map has to be deleted.

  • If there are many maps already initialized, one has to delete the records corresponding to the incorrect map via custom SQL script or procedure.

  • If there is only one map initialized and saved in the database, one can truncate all the data and also restart all the sequences for primary keys with the following command:

                TRUNCATE map, area, entry_node, path RESTART IDENTITY CASCADE;

Login button not working

First of all, make sure the AllInOne folder contains some map file. It might be the case that the first time the visualization application is loaded in a browser, the login button does not work and an error is thrown in the browser console. This is quite likely to be a problem with loading the webapp resources, thus we advice the webapp to be reloaded with disabled cache option.

• in Chrome this is possible by opening Developer Tools going to Network and enabling disable cache checkbox

NOTE: if this solution does not work, it is possible that a software is blocking the WebSocket connection between the webapp and the SmartFox server.

Docker

To run the server in docker container. Follow the steps:

1. cd into directory ./docker/
2. Build the docker image using build.sh
3. Build the docker image for RMI server. Instructions can be found in the rmi server repo.

Start the server with:

docker-compose up

or, to run in the background:

docker-compose up -d

Then the smartfox admin panel can be accessed with username sfsadmin and password sfsadmin under:

localhost:8080/admin

The visualization service can be accessed under:

localhost:8080/visualization

There is one last thing to do before visualization service is available. Login to the admin panel, go to server -> webserver and activate the websocket service. Click sumbit button and restart button. After that, the visualization service is good to go.

Although this smartfox image is currently set to use SQLite as default database, one can also use Postgres as database. To do so, follow these steps:

1. Build the postgres docker image using ./docker/postgres/build.sh
2. Follow step 3 of Server configuration above, when it comes to Connection string setting, instead of using jdbc:postgresql://localhost:5432/mmosim, use jdbc:postgresql://postgres:5432/server as Connection string.

Additionally, if you just need the postgres database and find it complicated to setup, you can use the following command to start a postgres database:

$ docker-compose up postgres -d