Rework: HOFFNUNG 3000


#1

I’m working right now on updating the HOFFNUNG 3000 plattform (short introduction here: https://edgeryders.eu/t/introducing-blatt-3000/9421/5) to make it accessible and useful for more people and organizations. I will use this thread here to post a little bit about my reasonings during the programming and rewriting phase :slight_smile:

The HOFFNUNG 3000 codebase is organized in a .git repository and hosted here: https://github.com/adzialocha/hoffnung3000. The main work was done in summer 2017 for the festival in Berlin which used that plattform for the first time.

The overall rework will consist of many sub-tasks I will document here in separate sections. You can find the PR for the whole thing here: https://github.com/adzialocha/hoffnung3000/pull/37


Introducing: BLATT 3000
#2

I. Housekeeping and first thoughts on configuration

Already back then we decided to offer some sort of customization, so other people can change the behaviour of the platform for their own purposes. While development we also focussed on an easy way to deploy the application (at this time we used Heroku).

As I am revisiting the code some time later I’ve realised that still there are many things hard-coded which are too specific for the usecase in Berlin and also some design choices in how to handle the customization are not completly right.

First I decided to update all dependencies of the project, as the last update is more than one year ago many things changed ( sequelize database orm), got deprecated or even removed (react-router-redux for replaced by connected-react-router).

While this update required touching upon basically every layer of the application, it was good to get in touch with all parts of it again. I used the chance to improve some small things, like debug logging and better linting rules.

Also I changed the build system from gulp to webpack, as we had long build times during development (>30 seconds per build) in the end. With webpack it requires as much time for the first build, but later on its less than a second, which makes development much more convenient of course. :slight_smile:

As I changed the build system I realized that some platform configuration is already happening in the asset building process (like setting the Title of the page in the generated index.html). At this point some configuration was handled through an .env file, which was alright in the past but looks pretty ugly to me now. In the beginning we’ve decided to decouple the webapp from the server strictly, which meant that the whole app was hosted as a static page, nothing was rendered by the server. This was why we had to move some configuration-related tasks already in the building of the application assets.

I decided to move all configuration into a database and decouple system related / low-level configs (database credentials, API keys etc.) from high-level configuration (Title of the page, Dates of the festival etc.). This has to be done in a later step, but as a first preparation I decided to render index.html file server-side and not in the asset building process.


First "housekeeping" steps for upcoming improvements and changes to the codebase:

    * Render _index.html_ via `pug` on server-side for easier customization
    * Change build system from `gulp` to `webpack`
    * Update deprecated `sequelize` methods
    * Update all dependencies and asset building pipelines (Babel, scss etc.)
    * Replace outdated `react-router-redux` with `connected-react-router`
    * Introduce `LOG_LEVEL` env variable for more control over logs, improve `winston` logger

Changes related to this you can find in this PR: https://github.com/adzialocha/hoffnung3000/pull/36


#3

II. Configuration of the backend

As already mentioned I decided to move all high-level configuration into the database, which required a new config data model.

I replaced all configuration critical parts of the backend with a small config wrapper which fetches the configuration value from the database instead of relying on environmental values.

The backend is configured through the database now, I will look into the web application now, as this requires some more changes (the application has to fetch the config through a new API endpoint).


Changes related to this you can find in this PR: https://github.com/adzialocha/hoffnung3000/pull/39


#4

Really nice reworking! Makes things easier, for sure.


#5

III. Configuration of the backend - Updates

The application is now fully configurable. I decided to leave all critical and secret API / Access etc. keys as environmental variables and move all high level configuration into the database. This enabled me to create a convenient new admin panel for configuration. Many features can also be turned on and turned off in the new admin interface:

I stumbled upon a lot breaking code due to the changes, but mainly due to the last dependency updates.

I’m considering making AWS S3 storage for image uploads optional, to make local deployment easier for some lightweight usecases.

This - and the feature switch for the anonymization of users are the last two changes before the customization PR is ready for merging.


#6

It’s looking great. Keep on truckin’! :slight_smile:


#7

IV. Improvements and last touches

I finished the work on the customization and went one with final touches and ideas I had for the platform. Some things were rather necessary, for example moment-js exploded the build size with Webpack (had a workaround with gulp before), so I decided to move to the (anyways better) luxon library which is relying on a new timezone API instead of importing all definitions inside the module.

Also there was some time left for optional changes, like:

  • Make AWS S3 an optional dependency, the NodeJS application just uploads everything into a local folder by default (this is interesting for people who want to run it on an own server and dont have an AWS account)
  • Don’t use GoogleMaps API but Leaflet with OpenStreetMap instead (now it is not needed to register an account with Google before for making the platform run)

There were many additional fixes and patches raining in during development, but it all came to a very nice production-ready state again.


#8

V. Merge and documentation

The technical part of the rework is mainly done (will do some more tests before but so far everything works again).

I also had a look into the related side-projects, like the gif-stream and gif-stream-client and updated them:

Soon I will merge the rework branch :sunny:

Next steps will be the creation of a nice and simple manual and documentation on how to setup and use the platform. I decided to make a simple static page which will be hosted later on hoffnung3000.de.


#9

VI. Handbook

The documentation was moved inside the hoffnung3000 repository in a docs folder (using jekyll and github pages for hosting).

I finished the installation guide, including a very simple Heroku setup which involves just clicking one button :smile:

https://hoffnung3000.de/


#10

Great! Is that then the last piece, or is there still something else left to do?


#11

Thats the last piece :sunny:


#12

@adz, I’ve tried setting it up on Heroku here: https://urbanburn-3000.herokuapp.com/
For some reason, it won’t let me pass through the last stage to register as a participant.

Edit: Figured registration out, had to set price to 0 if not using payments

There are other problems too. Image upload doesn’t work even though I’ve set the S3 variables and the calendar page view seems to crash now for some reason after I tried creating a second event. Any ideas on what might be going on?


#13

Great, just wanted to answer the price has to be set to 0 :slight_smile:

Did you set the permissions right for the S3 bucket (eg. policies)? You can have a look into the server logs to figure out exactly whats going on (LOG_LEVEL env var has to be set to verbose to get everything).

Did an error occur during creation of the event (S3 etc.?), it could be that you stored an invalid event without slots in the database which broke the calendar - not nice, the database structure should stay consistent. I can build something in to prevent these things to happen in the database, but the problem should be something else. My tip is to fix S3, wipe the database and try it again!