Issue with Edgeryders form

owen · July 28, 2020, 11:16am

Ok, the final block I have now is with getting the API key of an existing user - so the form can be completed without creating a new account.

I have implemented a process described here by @matthias and deployed it to https://webkitsandbox.netlify.app

The user is directed to the SSO login and returned to the page, which then makes a call to https://communities.edgeryders.eu/multisite_account_api_key.json.

My understanding is that after the login, a cookie is stored and the call should return the logged in user’s API key. However the data returned (which is displayed on the page after the redirect) does not do this, it contains only the HTML content of the communities site.

I have copied the request as a CURL command here:

curl 'https://communities.edgeryders.eu/multisite_account_api_key.json' \
  -H 'Connection: keep-alive' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_16_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36' \
  -H 'Origin: https://webkitsandbox.netlify.app' \
  -H 'Sec-Fetch-Site: cross-site' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://communities.edgeryders.eu/' \
  -H 'Accept-Language: en-GB,en-US;q=0.9,en;q=0.8' \
  --compressed

Any ideas - @Daniel or @matthias ?

matthias · July 28, 2020, 11:51am

But in the curl command you show above, that cookie is not sent to the server. And that would be the problem you have.

owen · July 28, 2020, 1:08pm

There is an issue with getting the cookie in the Vue front end.

I can see they are stored in the browser:

However calling document.cookie only returns the test cookie I set on the page load, and none of the others.

I will need to test this behaviour with other cookies as it may be an issue in Vue - but it appears cookies from other domains are not accessible for some reason.

owen · July 28, 2020, 1:11pm

@matthias I think I found an answer?

From http://en.wikipedia.org/wiki/HTTP_cookie:

Cookies are not directly visible to client-side programs such as JavaScript if they have been sent with the HttpOnly flag. From the point of view of the server, the only difference with respect of the normal case is that the set-cookie header line is added a new field containing the string `HttpOnly’:

Set-Cookie: RMID=732423sdfs73242; expires=Fri, 31-Dec-2010 23:59:59 GMT; path=/; domain=.example.net; HttpOnly

When the browser receives such a cookie, it is supposed to use it as usual in the following HTTP exchanges, but not to make it visible to client-side scripts. ~~The HttpOnly flag is not part of any standard, and is not implemented in all browsers.~~

Update from 2017: a lot of time had passed since 2009, and HttpOnly header flag is became a standard, defined in the section 5.2.6 of RFC6265, with the storage semantics described in the same document (look for “http-only-flag” throughout the RFC text).

There is no way to access anything about the HttpOnly cookies from “non-HTTP” APIs, e.g. JavaScript. By design, neither reading, nor writing such cookies is possible

matthias · July 28, 2020, 1:24pm

Did you make sure the Discourse server indeed sent a header “Set-Cookie: ... ; HttpOnly” to set the cookie? You can see that when analyzing the server’s response in the browser’s web developer tools.

owen · July 28, 2020, 1:26pm

The response header is as follows:

Set-Cookie: destination_url=https%3A%2F%2Fcommunities.edgeryders.eu%2Fmultisite_account_api_key.json; path=/

So it looks like this isn’t the issue

Update: yes it does, taken from communities.edgeryders.eu (emphasis added)

Set-Cookie: _forum_session=K1dWbUhxS012SkhhZkRhbkl5alQ2cDFqT3dMZTF0NzNkdzFSNFlicVVBeXV0THpkd3dFZDhXQVJiRWtnV3U5Qk5QaXplVWJEQTJvVTNQeVlDcHZEVkw1cDRyNno4RS9UR0VaTEZGYUlyRzl5SzRPSU1IQTNubEJGVXVRaVZrKzhPdWpJODRGT3NWckZadHFlM3lSSm1sQmpYL3hua2hid21TS3pjc3JSNzFXRG9VWTRBQ24wUkVMekZHbVNvVnJSLS1IN042bWRFKzg3azJYQ2kwYUdoWEJBPT0%3D--8d82218592a0adea55061593b34806e5dbb67836; path=/; secure; **HttpOnly**; SameSite=Lax

matthias · July 28, 2020, 1:36pm

In the Firefox web developer tools, the cookie list shows a column “HttpOnly”, and that is true for the _t cookie, which is the relevant one for login. (The _forum_session is for something else.) So yes, the cookie uses HttpOnly

Could you try setting HttpOnly to false for that cookie (works by editing the column in the cookie list, at least in the Firefox web developer tools). And then see if it solves the issue with your script.

owen · July 28, 2020, 2:00pm

Hm I disabled HttpOnly in Chrome, which has a similar interface to Firefox for this:

I reloaded the page and no difference there, so something is up with the cookies set from the communities site, but I’m not sure exactly what it is.

more discussion on this here

matthias · July 28, 2020, 2:05pm

I had expected that the relevant cookies will be automatically appended by the browser to any requests sent to communities.edgeryders.eu, including when sending the requests by JavaScript.

But I didn’t have to deal with cookies in cross-site scripting applications before. Maybe these cookies are just not appended to the request by the browser in such a case? And also not visible to scripts because they belong to a different domain? (Ah, actually they might not be accessible because now your script is on a domain that is not an edgeryders.eu subdomain. You can try.)

Probably you’ll have to read up on cookie handling in cross-site applications to figure out what’s not working right now. Sorry for the mess …

owen · July 28, 2020, 2:09pm

I’ll do a bit of reading and see what I can find out, but from a brief glimpse on StackOverflow there seem to be multiple issues with (1) httpOnly cookies and (2) sharing cookies across domains…

matthias · July 28, 2020, 2:12pm

There is the so-called Discourse User API Key mechanism to authenticate external applications for using somebody’s user account in Discourse. We did not use it before because authenticating them is complex given our setup (login to communities.edgeryders.eu, then login to edgeryders.eu, then confirm app access, then return). But if nothing else works, we should try that because it’s the solution prepared for these use cases.

owen · July 28, 2020, 2:21pm

I guess you probably know which solution is more ideal… if that authentication process is only required once per user, it wouldn’t be so bad - although perhaps not intuitive to a user.

another idea would be to have a small server side application with a master API key to create posts on behalf of a user, something like this would not be too difficult to implement:

user logs in via SSO and is redirected to site
site has account user ID, calls small server app with ID parameter
server app posts on behalf of user matching account ID with master API key

it might be another security risk and understand you’re not a fan of more middleware, but it could be the fastest way to get a working solution.

daniel · July 28, 2020, 3:34pm

We found this plugin which looks promising and might offer a simple solution to the problem. I’ll install it and check if it works for our requirements.

owen · July 28, 2020, 4:23pm

@Daniel This looks really good - by means of JWT, could a user potentially login from the front end application without redirecting to communities.edgeryders.eu?

daniel · July 28, 2020, 4:25pm

Yes, this should be possible.

matthias · July 28, 2020, 7:32pm

Ok, so here’s the solution: @owen, please use the off-the-shelf Discourse User API Key mechanism. Use it to let the user approve access of your application to their edgeryders.eu user account. Which means, the URL to forward them to for approving the application’s access is NOT https://communities.edgeryders.eu/... but:

https://edgeryders.eu/user-api-key/new

This is the default mechanism provided by Discourse for authenticating external applications. We should not try to roll our own alternative for this, because we’d have to maintain that for years, and we already know that all authentication related customizations easily break during Discourse updates. We spent three hours looking for other alternatives like JWT based authentication, but there is no ready-made solution for Discourse anywhere. So we’ll go with what Discourse provides out of the box.

As a result, you’ll have three cases:

If the user does not yet have an edgeryders.eu account, your application uses our custom multisite_account.json API endpoint as before.
If the user as an edgeryders.eu account and is logged into it, your application forwards the user to https://edgeryders.eu/user-api-key/new for approving access to their edgeryders.eu account. That takes only a single click, and then the user is forwarded back to your application.
If the user as an edgeryders.eu account but is not logged into it, your application also forwards the user to https://edgeryders.eu/user-api-key/new. From there, the user will be forwarded to communities.edgeryders.eu for login, then back to approve the application, then back to your application. This is obviously annoying, but will happen only in 5% of cases as people typically stay logged in.

You can (and should) save the resulting User-Api-Key value in offline storage into the user’s browser. That way, authenticating the application has only to be done the first time the user uses it.

owen · July 28, 2020, 9:30pm

Thanks @matthias, for the help and work - I will give it a go tomorrow morning.

just to be clear - when I forward to https://edgeryders.eu/user-api-key/new - am I including a url parameter of some kind? how does this work exactly for the redirect…

update - ok just read the documentation page.

matthias · July 29, 2020, 10:01am

The User API Key mechanism will not be simple to implement on the client side. But after looking around long enough, it’s the best solution for people holding an account. So take your time to get this right, and let nobody stress you about it Also, you might find a JavaScript library that already implements the client-side functions of this API key mechanism.

owen · August 4, 2020, 10:21pm

Hello @matthias - I’m still in the process of working on the login function, but I have implemented a directory for the forms with support for different fields.

Could you add these domains

to the whitelist for both edgeryders.eu and http://communities.edgeryders.eu/ ?

matthias · August 6, 2020, 11:57am

Sure. Both done now.