Bounty for developing Edgeryders form interface

gdpelican · October 13, 2019, 8:40pm

Alright, can confirm this will work for our use case; I do have to add a line (via this plugin) to PostsController to allow it to skip CSRF checking via the api:

skip :verify_authenticity_token, only: :create, if: :is_api?

I’ll push that up soon.

gdpelican · October 14, 2019, 5:43am

Cool, I’ve been successful in making posts from the vue app to a local discourse instance with that plugin installed; I’ve pushed that plugin up to GitHub - gdpelican/permit-api-cors: Allow a Discourse instance to accept Api-Key in the CORS settings

Here’s a screenshot of an example post (open to feedback on title / formatting / etc):

One thing I wasn’t clear on was whether we are posting all of the responses in a single topic, or each response gets its own topic?

Next up: There’s CORS trouble with hitting the multisite endpoint as well (obviously 'localhost:8080 isn’t a great thing to put in your list of accepted origins), but I’m hoping I’ll be able to install that plugin locally as well and get an end-to-end test going soon.

hugi · October 14, 2019, 8:34am

@natalia_skoczylas, what’s the intention?

Nskocz · October 14, 2019, 9:03am

They get their own topics - they will be collected in this category, right now under development, but soon to be open to everyone

https://edgeryders.eu/c/rethinking-retirement-academy-of-life

gdpelican · October 14, 2019, 6:46pm

Hey @matthias, could you confirm or deny for me that users created via multisite_account.json are set to ‘active’ or not? They’ll need to be based on this line in Discourse:

github.com

discourse/discourse/blob/main/lib/auth/default_current_user_provider.rb#L115


      
            user = nil
            user = User.find_by(id: uid.to_i) if uid
            @env[CURRENT_USER_KEY] = user
            return user
          end
          
          request = @request
          
          user_api_key = @env[USER_API_KEY]
          api_key = @env[HEADER_API_KEY]
          
          if !@env.blank? && request[PARAMETER_USER_API_KEY] && api_parameter_allowed?
            user_api_key ||= request[PARAMETER_USER_API_KEY]
          end
          
          api_key ||= request[API_KEY] if !@env.blank? && request[API_KEY] && api_parameter_allowed?
          
          auth_token = find_auth_token
          current_user = nil
          
          if auth_token

I believe there’s an option in the discourse api to pass active or not when creating an sso user, but I’m not sure if we’re using it or not:

github.com

discourse/discourse_api/blob/main/lib/discourse_api/api/users.rb#L55


      
                  :theme_ids,
                  :user_fields,
                )
                .to_h
            put("/u/#{username}", args)
          end
          
          def update_username(username, new_username)
            put("/u/#{username}/preferences/username", new_username: new_username)
          end
          
          def update_trust_level(user_id, args)
            args = API.params(args).required(:level).to_h
            response = put("/admin/users/#{user_id}/trust_level", args)
            response[:body]
          end
          
          def create_user(args)
            args =
              API
                .params(args)

matthias · October 14, 2019, 8:12pm

Right now it’s done this way:

So the new “master account” over at our SSO provider site communities.edgeryders.eu is not set to active, but the new edgeryders.eu account is set to active. Due to that, you should be able to use the returned edgeryders.eu API key immediately.

gdpelican · October 16, 2019, 9:01am

Alright, I think this is ready for an initial test run; I’ve gotten it posting to a local discourse instance using the multisite-accounts plugin and mimicing the setup we’ll have in production as closely as possible.

Three things will need to happen on the edgeryders.eu instance:

Set the DISCOURSE_ENABLE_CORS env to something truthy if it’s not already
Add the domain hosting the vue app to the cors_origins site setting on the admin panel
Install the permit-api-cors plugin

Steps 1 and 2 will need to be repeated on communities.edgeryders.eu as well.

Here’s the env we’ll want to set for the vue app:

VUE_APP_DISCOURSE_USER_URL=http://communities.edgeryders.eu/multisite_account.json
VUE_APP_DISCOURSE_AUTH_KEY=<auth_key>
VUE_APP_DISCOURSE_DOMAIN=edgeryders.eu
VUE_APP_DISCOURSE_TOPIC_URL=http://edgeryders.eu/posts.json
VUE_APP_DISCOURSE_CATEGORY=343 (<-- this will be changed to 339 for the actual category)

Let me know what snags you run into, and if it successfully gets going, what additional design / styling / functionality work should be done?

matthias · October 17, 2019, 12:46pm

I’ll get this done as soon as I can carve out the time. Could still be 2 days from now since we’re all squeezed right now to finish some website and communications materials for parallel projects …

gdpelican · October 24, 2019, 9:20pm

Hey @matthias what’s the ETA on this one? I’m aware of the tight deadline and want to make sure we’re not in a position where the conference is looming and things aren’t working properly

matthias · October 24, 2019, 9:25pm

Thanks for your concern! ETA was yesterday, but the E was wrong. Now I’m having a bit of time again, so let me try and get it done right now …

Are you around for the next 1-2 hours in case I have questions about installing your work?

gdpelican · October 24, 2019, 9:38pm

For sure, fire away

matthias · October 24, 2019, 10:29pm

The env file naming in your Vue.js application is automatically including the deployment environment? So should I name the file .env.production probably? If so, how does it get to know about what deployment environment it’s in?

(I’ve set up the Vue.js application on https://bio26testing.edgeryders.eu/ now. Not sure yet about the env part though. Anyway, now to finishing the changes to Discourse …)

gdpelican · October 24, 2019, 10:44pm

I believe vue will take the values in .env.[ENV_NAME] over the defaults, so yes a .env.production file should work.

It looks like the env is getting loaded alright now, although it seems we need https for the protocol on the urls rather than http

VUE_APP_DISCOURSE_USER_URL=https://communities.edgeryders.eu/multisite_account.json
VUE_APP_DISCOURSE_TOPIC_URL=https://edgeryders.eu/posts.json

matthias · October 24, 2019, 10:51pm

Thanks for having a look. Made the switch to https:// URLs now.

matthias · October 24, 2019, 11:44pm

Ok @gdpelican … we’re getting closer to success here, but it’s still a bit of way to go I followed all your instructions, and:

Now when testing https://bio26.edgeryders.eu, in the last step I get a red error message “TypeError: Failed to fetch” on screen, and “Failed to load response data” in the web dev tools’ “Network” tab for the request to https://communities.edgeryders.eu/multisite_account.json. I guess that means my CORS settings on communities.edgeryders.eu are not yet accepted.

So far I have:

Created a file app.yml in the root of the Discourse project file tree and put in the following:
```
env:
  DISCOURSE_ENABLE_CORS: true
```
(We’re running Discourse standalone, without Docker. So we don’t have the app.yml file by default.)
Added the following entry the “cors origins” setting in the communities.edgeryders.eu admin backend:
- https://bio26testing.edgeryders.eu
Restarted the webserver that is serving communities.edgeryders.eu (Puma in our case).

Documentation about where to put the DISCOURSE_ENABLE_CORS setting is scarce. Did you find something that I missed?

matthias · October 25, 2019, 12:22am

Logging off to sleep now I’ll look into this again first thing in the morning.

gdpelican · October 25, 2019, 1:21am

I was going off of the message that comes along with the cors_origin site settings:

There’s a touch of documentation in the code about this, although I’ll have to admit I had to dig for it as well, maybe this helps?

github.com

discourse/discourse/blob/ed00f3530621d61a272b237746b8bbb1eaea7998/config/discourse_defaults.conf

#
# DO NOT EDIT THIS FILE
# If you need to make changes create a file called discourse.conf in this directory with your changes
# On import this file will be imported using ERB
#

# Discourse supports multiple mechanisms for production config.
#
# 1. You can do nothing and get these defaults (not recommended, you should at least set hostname)
# 2. You can copy this file to config/discourse.conf and amend with your settings
# 3. You can pass in config from your environment, all the settings below are available.
#    Append DISCOURSE_ and upper case the setting in ENV. For example:
#    to pass in db_timeout of 200 you would use DISCOURSE_DB_TIMEOUT=200

# All settings apply to production only

# connection pool size, sidekiq is set to 5, allowing an extra 3 for bg threads
db_pool = 8

# ActiveRecord connection pool timeout in milliseconds

This file has been truncated. show original

matthias · October 25, 2019, 9:54am

Thanks, that helped! The setting was even already in the discourse.conf file, waiting to be enabled. Just not named DISCOURSE_ENABLE_CORS …

Now the user is created alright, on both communities.edgeryders.eu and edgeryders.eu. The final call to https://edgeryders.eu/posts.json still fails, indicating another CORS problem. On this now …

matthias · October 25, 2019, 12:47pm

Ok, I got past the CORS problem … means, the browser is sending the final posts.json request now. Also, when I simulate a CORS pre-flight request with curl, I get the expected response incl. Access-Control-Allow-Headers: […] Api-Key, Api-Username. Which means your plugin is running, as it’s adding these two values.

So far, so good. But I get a “403 Forbidden” response for the actual POST request to posts.json.

tl;dr: That must be a bug in the version of Discourse running edgeryders.eu and happens for all authentication with an Api-Key: header. It does not happen on our Communities sites (listed in the top-right menu “Communities” above), which is running a very recent Discourse version. Solution: I’ll make a test environment on a Communities sites, and get our Discourse guy to update the edgeryders.eu installation upon his return. If he’ll not get back in time, we’ll also have to place @natalia_skoczylas’s responses into a Communities site at first. Really don’t have the time myself right now for a full Discourse update

@gdpelican please have a little more patience while I create a testing environment on one of the Communities websites …

Details about investigating the error

But I get this as a response to the posts.json query:

status code: 403 Forbidden

response content:

{
  "errors": ["You need to be logged in to do that."], 
  "error_type": "not_logged_in"
}

server access log entry by Discourse:

Started POST "/posts.json" for [IP address] at 2019-10-25 12:50:48 +0200
Processing by PostsController#create as JSON
Parameters: {"title"=>"…", "raw"=>"…", "category"=>"339", "post"=> {"raw"=>"…"}}
Completed 403 Forbidden in 1ms (Views: 0.1ms | ActiveRecord: 0.0ms)

This happens with all requests that use Api-Key: header for authentication, regardless of the API route / endpoint. Even a simple GET request to an access protected topic is failing on edgeryders.eu:

curl 'https://edgeryders.eu/t/about-the-survey-tests-category/11116' -H 'Api-Key: [redacted]' | less

Things that are not causing this:

Wrong API key. Because the same API key works, for GET requests, when appended as a GET parameter.
Reverse proxy setup. Our Discourse site is reverse proxied by Apache, but the same behavior is seen when accessing the Discourse Puma server directly at http://edgeryders.eu:9292 (note: HTTP only; also note: works with curl, but does not show anything in the browser as JS and other static assets are served by the Apache reverse proxy). So I can exclude that this is because the Api-Key: header would get lost during the proxying.

The error, as tested with the above simple curl GET query for an access protected topic, does not happen on our more recent Discourse installation running the Communities sites. So it must be due to a bug in Discourse, and needs an update of the edgeryders.eu installation.

matthias · October 25, 2019, 11:29pm

That’s done now, and lo and behold, it works Great work @gdpelican, and a ton of thanks, also in the name of @natalia_skoczylas.

I created one test topic via the forms software, and it’s preserved here (with my test responses showing as ######…):

https://community.trustinplay.eu/t/rethinking-retirement-response-by-test02-ansorgs-de/628

Our testing setup is now like this:

Edgeryders Forms software: https://bio26testing.edgeryders.eu/
target Discourse site: https://community.trustinplay.eu/
target category: Academy of Life, category_id 16

Necessarily, the target category is public, as the new users are not members of any groups that would allow them to post in a hidden category. Due to this, let’s be a bit careful when testing to not spam everyone else on the platform. (Just a bit careful. 30 posts are still ok, just not 300.)

Also when testing, please use consistent naming for the e-mail addresses and also so that it expresses that these are for testing purposes only. That will help me delete all test users in the end. The e-mail addresses do not have to exist, so you can for example use a scheme like this:

testuser100@example.com
testuser101@example.com
testuser102@example.com
...

The live setup will be similar, just that we’ll rename bio26testing.edgeryders.eu to bio26.edgeryders.eu and maybe move it from our server to Netlify hosting.

Next step: I added a few issues on Github I found during my testing, for @gdpelican to work on. @natalia_skoczylas should do thorough testing now and report everything that should be changed, either in this thread or as a Github issue. No need to report spelling mistakes etc. though, this is our part to adapt when we install the live version. Also @nadia may want to have a look to see how a similar form can be used for the H2020 projects.