Redezem's Blog

Privacy? In this economy?

Privacy? In this economy?

Have you ever been on a trip somewhere outside your home countries borders? Don’t ask, just pondering for reasons involving being on a plane. For those of us that have, you’ll be well aware that other cultures tend to have… well… different things they consider acceptable.

This is also the case in networks. And funnily enough, the kinds of networks you tend to find in travel. For whatever reason, airports treat their wifi networks like the only users are the sort of people that are liable to shut off the water supply. Now sure, the average airport has a veritable cornucopia of terribly dangerous things, and the average person does sort of think that a hacker is essentially a powerful modern dark wizard. But as someone who actually knows how this stuff works, if the public wifi is even on the same hardware that the critical systems are on, you have seriously failed in your design. Sure, most if not all public wifi networks are sharing hardware with corporate networks, but I also don’t really consider the average network architecture to be particularly good. Long story short: This tends to be done out of ignorance and fear, rather than any real meaningful reason. Hackers can’t really make fuel tanks explode, and most people aren’t the sort of person that can successfully VLAN hop on a dime and while sleep deprived anyway.

Now, as for why I’m wasting time talking about airport wifi networks… It should not be entirely surprising that I’ve been on a trip. Internationally, too. And, while on trips, I tend to use a VPN. This is almost always the thing an airport is trying to block, mostly because it obfuscates traffic to the internet in a way that they can’t track. Or at least that’s what they’ve been told. I’d like to imagine it’s a security engineer telling whoever pays for the firewall, though it’s just as likely the salesman. Anyway, the theory would be that VPN traffic could hide something scary. For the same reason they also tend to block things like Github or anyone that might be peddling in gasp, hacker tools.

Anyway let’s talk hacker tools.

Getting Out Of The Average Network

Let me just start by addressing the obvious question: Why? Why the hell spend time at all to bypass someone’s cool smart firewalling solution?

And I mean, that’s a good question. Why does the average person buy NordVPN but for the chance to watch Netflix in other countries? I would hope the “Stop hackers watching your wifi traffic” line gets some sales, that’s what I’d be worried about. I’d be worried about it because the person watching your wifi traffic is me. You have no idea how many pcaps I have of random public wifi traffic. Every now and then I can find something interesting about how the owner of the wifi network makes the whole thing work… but mostly it’s just encrypted web stuff. Don’t worry, I’m almost certainly not watching you watch youtube, thanks to Edward Snowden the internet is so heavily encrypted that none of us are getting at your subscriptions that way.

That said, there is someone perpetually watching your behaviour in the free wifi zone, and that’s the people running the wifi. They might not be able to determine what you’re looking at on youtube, but thanks to the inherent limitations of TLS, they can see exactly what website you’re dialing in to.

Firstly, you may think “hey man, I use DoT or DoH instead of raw DNS, I should be safe right?”

So you don’t know what DoT or DoH is

Right well, okay I’d better explain myself.

DNS is a great tool. It lets you type in a sane name (say, “blog.redezria.net”) and the computer will figure out an IP address for you (say, 127.1.1.2). It does this by asking your DNS nameserver (which I can almost guarantee is set up to be your local wifi router). Your DNS nameserver probably doesn’t know what the sane name is (it is a computer after all, and probably only knows about things local to it… unless it’s a pihole and then it feigns knowledge about a lot of advertising servers and they’re all in the room down in the basement behind the sign “beware of the leopard”) and asks its DNS nameserver, and so on until somebody has a clue and sends the IP address back down to your machine.

Very cool and good and you may have heard of this before. However, what you might not know is that DNS is entirely in plain text. No encryption, no obfuscation. Any idiot between you and the internet, or within wifi range, or that owns a DNS server between your router and whoever knows where the website you want to go knows where you want to go. Makes you think who else knows what web pages you visit in incognito mode, doesn’t it?

Thankfully there’s a solution. Actually two solutions. DoH is a simple one where DNS requests are actually performed over the HTTPS protocol. This is pretty clever, as most of the traffic you’re gonna be generating on a daily basis is HTTPS, cause that’s what the web works on. Downside is that you have to make a single plain DNS request to find the address of the website that runs the DoH server (as it’s a very cool and complex website at the end of the day). DoT is the same idea, but the DNS request is tunnelled in raw TLS. Now TLS is the same encryption standard as HTTPS, it just means that you don’t have to do a DNS look up as you’re not actually looking for a website, you’re just using a random TLS service on some server somewhere. It’s slightly more direct… but honestly there’s no real difference.

That’s Cool And All But Your Computer Is Stupid

So, you’ve gone and used DoT or DoH to make sure no pesky observer can watch your web browsing. Very cool! Just sucks it doesn’t help at all.

Yeah so fun fact, TLS is stupid as hell because people keep “improving” the internet. You’re going to love this one, absolutely laugh yourself stupid. Okay so: TLS works on certificates, right? And each certificate is for a specific website or set of websites. The idea is that if you get the certificate from a server, that tells you who you are talking to with some certainty. Problem is of course that the internet is too big and has too many people on it for one server to be youtube for example. So naturally, there are many servers that say they are youtube. That’s fun and cool of course, but most of us don’t have like fifty billion random servers. What actually happens is that people rent services that have like fifty billion servers and get that to distribute their website. We’ve got a few of these Content Distribution Networks (CDNs), one example being Cloudflare… who, speaking of, is the default network doing the DoH thing I talked about earlier. If anything can be called “Web Scale”, it’s Cloudflare. Honestly most websites you visit are on Cloudflare… Makes you wonder what they’re doing with all that DNS information…

Now, here’s the problem… if every third website you visit is Cloudflare, how does Cloudflare know which TLS certificate to use for each site? It can’t use the same one, cause that would be a gigantic security risk. This is an extremely hard problem, as it is essentially a chicken-and-egg style thing. If you ask for a certificate to do encryption, you need to tell the system which website you are asking for, and we don’t want to tell anyone where we are going before we are fully encrypted, otherwise what was the point of the DoT/DoH? The CDN can’t possibly know which certificate you need before you ask for it, and you don’t want to ask for it without certificate level security! What is a CDN to do?

Well what they do is just YOLO it and make you send the name of your intended destination in the Client Hello portion of the TLS message. Yep! It’s still in plain text, there was no point in securing DNS! And the best bit is that firewall operators know this, because there are monitoring tools that listen to all Client Hello messages to get the same domain information they were previously getting through the DNS traffic.

Getting back to the point

Gosh this is taking a bit. You can really tell I was on a 7 hour flight when writing this can’t you?

The only solution that will actually keep your browsing data secure is a VPN. And it’s gotta be a VPN you control too, NordVPN is keeping logs kids, sorry to break it to you. The reason all these wannabe panopticon network assholes are collecting what you’re looking at? Data, primarily for sale to data brokers, or to inform marketing directly. It’s Surveilance Capitalism again! Why did you think the wifi was free in the first place?!

Now I respect the hustle, I guess, but what I don’t respect is anyone telling me that I can’t use whatever I want. I’ve always had a bit of a “rules are for other people” bent when using computers, and if someone says “you need to let me watch where you go to use this network”, I’ll ignore them. If they then attempt to use technical capabilities to force me to submit, I get mad. Who are they to tell me what I can and can’t do?

I’d tell myself to get a grip, this is such a small deal, but hey, if I did we wouldn’t be having fun right now would we?

Anyway, in most cases all a network operator is going to do is block known VPN and proxy addresses. If you’re lucky, your VPN will have a couple servers too new for their block to know about them. In some other cases, they’ll block VPN ports, which means you just gotta use something like OpenVPN over port 443 (which is the HTTPS port). Most of your NordVPNs (yes that’s right, I’m Kleenexing the youtube advertised VPNs, cause they’re basically the same) are able to do this already, it gives you a little bit more resistance to being blocked.

If the network operator is less kind, they might kill any connection that seems to run for a long time. This will nuke any TCP VPN, and prevent you from using OpenVPN across 443. This is because web connections tend to be short lived. The downside is that this will probably break WebSocket applications on the web too, as they are long lived connections… but if the network operator is doing this they probably don’t know what a WebSocket even is, and probably don’t care either. The solution here is to just use something like WireGuard instead of OpenVPN. It’s a UDP protocol, so it doesn’t have any “connection” status that someone can make sense of.

Usually, you’ll face some combination of the above and will need to find a non-blocked server that supports WireGuard or something. I have a reasonably complex SD-WAN network infrastructure based on ZeroTier (shout out to ZeroTier, easily my favorite networking solution), and the rather resilient nature of the ZeroTier system means that most network blocks are only partially effective, and I can usually worm a Shadowsocks connection through to one of my home servers.

ZeroTier is what you’d call an overlay network, basically a complex VPN that does way more than just point to point connections. It lets you define full networks of remote machines that find and communicate with eachother. It’s also not based on any other existing VPN protocol, which means it’s not readily discernable at the firewall.

Shadowsocks is a SOCKS proxy protocol, which is to say it is an everything proxy instead of just a web proxy. Shadowsocks is special as it’s a TLS encrypted SOCKS proxy with a lot of obfuscation capability. It is part of the ongoing cat and mouse project to smuggle connections out of the Great Firewall of China. If you’re good enough to penetrate the most sophisticated network monitoring system on earth, you’re probably good enough to poke holes in the average corporate public wifi.

Between the two of these, I’ve got a pretty good and pretty secure means by which I can VPN to my home network, where the bits are free and the only obsessive invasive monitoring is performed by yours truly.

Fear and Loathing in UDP

So there is a thing that I’ve seen a few times now that really makes things hard. It’s a bit of a scorched earth policy in that it pretty much breaks everything but web access… But it’s the sort of thing you do if you really want that juicy data or you really want to clamp down on freedom. Some networks just straight up deny UDP at the firewall in addition to everything else. This degrades newer web apps that use QUIC (HTTPS but in UDP, basically. It’s a friccload faster), games, voip… a lot of things. But it sure as hell breaks most VPN technologies. And as such, it also forces most users (even savvy ones) to either use the network as commanded or give up.

I learnt this week from personal experience that Hamad International Airport in Doha does this. I learnt from friends in Helsinki this week that most corporate networks in the Middle East tend to block UDP as a matter of course. Paranoia? Possibly… but it could also be surveilance with intent. And given how strict these places are, I would not be at all surprised that this had an evil edge.

At any rate, I will not be forced to be surveiled. As Rage Against The Machine aptly said:

Fuck You I Won’t Do What You Tell Me

Now for the fun stuff. How to breach basically any network’s monitoring system in ways that will be very, very, very hard to catch. You will need:

The Strategy

The best way to avoid being stopped here is to not be using any IP address that the system knows is a VPN server, while also using a protocol that is extremely common, in a way that appears completely inconspicuous. The only way an adversary could block you would be to either know it was you and block you specifically, or have to block such a broad amount of network traffic that the network becomes effectively unuseable.

The easy part of this is getting an IP address, any server with public internet access will do. That said, the adversary could figure out it was a VPN server and block it later. So if we wanted to be really smart we could be lots of IPs. If websites can do it using CDNs, why can’t we use the HTTPS protocol, and a CDN to make our VPN web scale too?

Why not Cloudflare? That way they’d have to block Cloudflare. Specifically, they’d have to block web traffic to Cloudflare, which is akin to blocking the internet. That sounds like a hard enough problem to make it intractable for our adversary.

First attempt: V2Ray

Turns out this is a technique used to bypass the Great Firewall of China, at least in the case of a single server. V2Ray is a protocol tunnelling technique that allows you to tunnel certain protocols through other protocols, such as websockets and web requests!

As this is a popular GFC holepunch, there are tools to set up a server for this already, as part of Shadowsocks. The trick is that Shadowsocks in general likes the idea of connecting directly between server and client. We want to go via Cloudflare, so some finnessing will be needed.

Cloudflare connections aren’t too hard. If you make Cloudflare your domain’s nameserver, you can create reverse proxy lookup tunnels that allow Cloudflare to distribute your website across their meganetwork. You can install a program called cloudflared as part of this, which is super easy to use.

Pointing cloudflared to the Shadowsocks server with v2ray enabled allowed me to confuse the absolute heck out of my Shadowsocks server with my browser. However, I ran into a major problem when I tried to point the Shadowsocks client at it. Namely that it didn’t work.

I didn’t even manage to get it to reach out. Even though Shadowsocks-v2ray’s documentation swears that you don’t need to include a server ip address, just a server domain name, in practice you do. I don’t know if this is a versioning issue, or if I didn’t understand what the v2ray plugin actually did, but there’s basically no easy way to make Shadowsocks do a domain lookup for me here. I think it’d be possible to get further along by using an actual Shadowsocks URL (starts with ss:// instead of something you’re probably more familiar with), but in hunting around for what I did wrong… I found a much better solution.

See, even if I got this working, v2ray would be running over a websocket. Yeah, that’s pretty difficult to tell apart from regular web traffic… but it’s still a long running TCP connection. As I said earlier, that’s gonna get caught and killed. What I want is to look like regular web traffic.

Attempt 2: Cloak

Cloak is a toolkit to allow network monitoring evasion against state actors. This is probably the coolest networking toolkit I’ve seen in a while. Basically, cloak is a meta-proxy that turns VPN traffic into web traffic in a way that looks and acts like web traffic. It does this by segmenting communications into chunks, and then sending the chunks as short lived HTTPS sessions… which is exactly what regular traffic looks like.

Bonus: Cloak redirects sessions that fail to do the super secret proxy handshake to other websites! It looks completely untoward.

Now as Cloak for all intents and purposes is just web traffic, we can just point cloudflared at it, and then point Cloak at my Shadowsocks server minus the v2ray plugin… and it should just work.

Now, it didn’t… but it wasn’t because it didn’t work. It didn’t work because Cloudflare stopped it.

The Weakness

Cloudflare stopped the connection because Cloudflare was unsure of which server Cloak’s client side was connecting to. See under the default settings, Cloak neglects to tell anyone what url it’s looking for. This is because it assumes it’s going directly to the server, so it wouldn’t need to tell anyone (which of course, is more secure).

Conveniently, Cloak has a CDN mode where you can tell it what website you’d like to go to. The idea is that you tell the CDN you want to go to a different unrelated site, and then when the CDN connection is actually made and encrypted, you tell it you’d like to go to a different site. This can work if the certificate in question covers both sites (you say you want to go “totally.legit.website” and then secretly go to “vpn.legit.website”, and both are covered by a “*.legit.website” certificate), but is not at all how Cloudflare works.

This means that there is a single weakness to this strategy. Client Hello filtering. If someone watches TLS Client Hellos with Cloudflare, they’ll see the name of my cool Cloak server, and if they can block the connection based on that, they can block my VPN.

The Good News

In what will be a boon for me, and the general cause of defeating tyrrany, there is a new trick TLS 1.3 is introducing. Encrypted Client Hello.

Basically, there are two encryption sessions: One between me and Cloudflare, and one between me and my server. In essence, I tell Cloudflare in my Client Hello “Hey dude I’d like to ask you about a website, but on the down low”, and once that TLS session is set up, then I ask for the actual server I’m asking for. This is not yet available everywhere, but will make this approach I’ve built nearly impossible to detect with regular tools.

This actually has some implications for my day job too: We do network monitoring and network detection at Hyprfire, based on some behavioural analysis magic. The use of Cloak would make it very very hard to detect malicious activity, and the use of Cloak over Cloudflare would make it nearly impossible.

And So

You’ll note I’ve not included the commands on how to do any of this here. Honestly, it’s because I’m tired, and because I figure half the fun of doing all of this is figuring it out. If you pop over to the Cloak website and spend an hour or so playing with Cloudflare access tunnels, it’ll be pretty obvious how to get this thing up and running.

Oh all right then, here’s Cloak: https://github.com/cbeuw/Cloak

But, assuming you can do some googling, and some work putting together the parts to make this work… you too can bypass network monitoring and censorship with impunity.

What I like most about the approach of using Cloudflare (or another CDN, whichever you use isn’t really important, they all do the same kinds of things) is that the CDN will restructure its internal connections to reduce latency to your device. CDN customers know that the speed a website loads has a direct impact on user retention, so CDNs will do everything they can to increase speed when possible. This means that the Cloak/Shadowsocks tunnel is surprisingly fast after a little bit of usage. That said, I wouldn’t use it to download too much… if this solution starts costing a ton of money it’s utility drops precipitously.

Now for the sad bit

You know what the worst bit about all this is?

It didn’t work!

I know what a disappointing payoff. After all that cool build up and the neat solution and everything! Ah well.

Full disclosure, this approach worked more than fine in the hotel network and on the Helsinki Vantaa Airport public wifi, so I’m 100% confident that if you can get the connection started, it’s essentially invisible to prying eyes. But, of course, there’s always something. And in this case, well… let me tell you what happened:

So, I get into Hamad, whack on the ol’ cloak, and immediately get my connection RST’d (after an unusually long- ie a few milliseconds- wait). Basically something at the firewall was straight-batting my packets right on back to me… but how? Was it my cool DNS name? Was there a security solution that was able to detect Cloak? My god, did they know who I was? Do they have agents listening to me on Discord?!

No, none of that as it turns out. Cloak actually includes an Encrypted Client Hello packet, even under default settings. Not sure why that’s the case, might be part of how it handles the initial handshake with the CDN or the Shadowsocks server… anyway it’s always there.

You know how I said above how ECH is basically hard mode for network monitoring? Yeah, who on earth wants that up in their network if they want to maintain total coverage? I am fairly sure based on watching network traffic for about 30 minutes that the Hamad firewall is just terminating sessions with ECH packets for the same reason they’re terminating UDP. Now at some future point, they’ll have to change… but for now that’s an effective defensive strategy against Cloak.

Long story short: I need a better tool and I only had 45 minutes between landing and boarding my next flight. Either I’ll find one by the time I fly on back, or I’ll write a new one. Good reason for another blog post anyway!

Anyway, I hope you enjoyed my rant, and hopefully you know a little more about evading monitoring on a network. Sure, it didn’t work here at this time, but the important bit is that nobody could effectively know what I was trying to do anyway. And hey, if the network you’re on isn’t as super-corporate as a mega-airport… well it’ll work just fine :)