Arduino is pretty popular among tinkerers community. Its a solid way of getting into the world of embedded systems as a beginner. Arduino community/ecosystem is quite big with a lot of tools/libraries/hardware components/literature/forums and what not. Everything is open-source, leading to quite a number of clones which are in some way or another better than an equivalent vanilla Arduino. These clones/derivatives mostly focus on better energy consumption, smaller footprint, cost reduction, integrated connectivity etc.
But nothing in the whole ecosystem, and I mean NOT A SINGLE ENTITY, from everything I listed above, addresses the fact that Arduinos can't be flashed remotely. You absolutely have to connect an Arduino to a USB host to do a "sketch upload". Now there are some ways to update Arduinos in a local network without a USB connection, but anything outside of it just seems impossible. You could, in theory, make your Arduinos accessible externally using port forwarding, or even install a host in your target network and control that instead. How about not doing that?
"Meh," some of you are saying right now, "nobody uses an Arduino for an application that requires remote flash update. There is Particle for that." Let's not get into that discussion because the question is not "What's a good Arduino replacement?". The question is "Why there are Arduino replacements?". There are times when you HAVE to use that one thing. I've seen a lot of hardware startups using Arduino as a prototyping platform, and then moving on to custom PCBs with Arduino cores. They would know what I'm talking about. Also, not to bad-mouth Particle products, but they aren't exactly Arduinos. There is nothing "deterministic" about them, if you are into that stuff. They are not without their fair share of bugs either.
Arduino sketch upload - behind the scenes
For those who don't know how a sketch gets transferred into an Arduino's flash memory, read on.
- You write C/C++ code in the Arduino IDE. All these source files are put into a single directory and are collectively called a "sketch".
- You press "Verify" in the Arduino IDE, which compiles your sketch into a "hex file" using avr-gcc. This hex file has all the information a "programmer" needs to put program bytes into their correct places in Arduino's memory.
- You press "Upload" in the Arduino IDE, which invokes "avrdude" to transfer hex file data through your computer's USB-serial port. On the other side, as soon as you press "Upload", the Arduino resets and its bootloader goes into a state where it is ready to receive hex file data from your computer.
- Arduino receives pairs of an "address" and a "byte-page" from your computer and actually writes them into an Arduino's flash. This process goes on until all the pages are written.
- The bootloader makes your Arduino jump to your sketch code and it starts running. The bootloader is still there somewhere, in case you need to repeat all of the above steps again.
The interesting thing to note here is that the bootloader, on a subsequent Arduino power cycle, needs some way of knowing whether to wait and receive next hex file data, or just jump straight to the sketch code.
Ethernet bytes == Serial bytes?
My question is: How hard can it be to use an Ethernet interface to receive the hex file data instead of the serial port? Arduinos have pretty basic but solid Ethernet functionality with those Ethernet Shields. Why can't it receive hex file data through a NAT as a client of a TCP/UDP server?
The answer is: it can! You just have to put some work into designing a bootloader which handles all of the Ethernet communication, along with other flashing steps. And then write a server that could serve ordered bytes that need to be flashed. Arduino has quite a big chunk of its flash dedicated for bootloaders. People tend to "save space" by miniaturizing bootloaders, foregoing the functionality they could achieve by not doing that. I'm sure a bootloader that occupies all of that space, and achieves remote flashing function is worth it.