Very excited about this discussion, and to hear about these developments in open source tooling! I’ve been away from FPGA programming professionally for several years now, but at one time I worked on some components of a project for H.264 encoding incoming camera feeds (GigE Vision) in real time. At that time I unfortunately had basically no prior experience working with video signals! I worked a little on the processing pipeline (demosaicing, filtering) and with the gigabit interfaces, and on control software for controlling DMA – streaming from Ethernet, to RAM, to cores, back to RAM for verification.
This was all on a Xilinx part (wanna say Virtex-5?) with a hard PowerPC on-die. Later I also worked on some low-power signal processing / data capture applications with some devices by Microsemi (defunct I think) and Altera (bought out by Intel) as well as a bunch of microcontroller programming. However I’ve never really done any board design, I know just enough electronics to read a schematic (sometimes) and break stuff (frequently). I’m excited that a project like this could push me to learn more about that but also moderately filled with dread by the words “very fine-pitch ball grid array”.
Here is some assorted thinking out loud and link dumps from the little bit of research I’ve done since finding this topic (and promptly signing up an account here).
FPGA hardware thoughts
-
“Flagship”-series / hardware-acceleration FPGAs (for Lattice I gather this is the ECP series) will commonly have a lot of “hard” peripherals on the silicon that you can instantiate from verilog - gigabit transcievers, memory controllers, some have complete processors on-die like the Memory Palace’s XC7Z010 SoC. Gigabit transcievers seem potentially useful for working with signals like HDMI, Ethernet, USB, even PCIe – seems awesome to me to have some high speed digital interface with a PC. This can be accompanied by a lot of complexity with drivers, etc. but seems like a really interesting possibility.
-
Many parts also have a DDR SDRAM interface, which can be very valuable for things like framebuffer applications, or if you want a CPU core on the FPGA. The ECP series chips have this.
-
I2C peripherals or USARTs with I2C/SPI modes are also sometimes available in silicon, depending on what application your FPGA targets. Looks like Lattice iCE chips have this. For either of these you can also potentially use / adapt some existing cores – the Lattice I2C controller reference design mentions it’s based on this one from opencores.org.
-
Lattice CrossLink devices have I2C hardware as well as MIPI D-PHYs which could be used for the CSI-2 camera interface. This looks to be a family of smaller FPGAs intended for video switching applications, but they have fewer fabric resources than the ECP chips, and the open source toolchain doesn’t support them at present.
-
I found this page from Lattice describing a framebuffer core which interfaces with a memory controller to talk to SDRAM, which is used for the actual frame storage. For the iCE series this is a big chunk of available fabric and for some chips might not leave enough to also have a CPU core if you wanted one.
-
Speaking of fabric resources, the closer you get to using the whole chip, the harder it often gets to route your design / meet timing requirements. In some cases you can use a floorplanning tool to manually provide placement hints to alleviate this, but this kind of thing can be challenging to deal with due to the constraints imposed by the PCB design and allowed chip I/O assignments. It’s worth keeping this in mind when deciding on a chip and estimating how much fabric you’re going to need.
-
Having a CPU, hard or in fabric, on-chip can be really nice for basically having programmable fast access to registers for controlling peripherals and stuff, even if a bunch of higher-level “application” type stuff is handled by a dedicated MCU chip. Serial communication and controlling peripherals all in verilog is a lot of work, and interesting but not near as enjoyable a use of FPGA coding time as cooking up high speed image processing pipelines.
-
Once upon a time, Xilinx sold analog video I/O expansion boards for their high end dev kits. Here is the user guide and the block diagram. It appears to have supported a load of different interfaces. Uses the ADV7403 decoder, which seems like a neat chip, I don’t know if there are more recent models with this kind of feature set.
Requirements and design thoughts
-
Accepting glitchy input is an interesting problem. I don’t know if anyone has heard anything about how well LZX TBC2 can tolerate glitchy composite inputs? Guess we’ll have to see once they’re in the wild. Might also be interesting to look into what hardware is used for the 1V I/O on Orion modules. One thing that’s interesting about decoding in fabric (if possible) is that any decoding you don’t need for some application, you can use those resources for something else.
-
On the video compression project we converted to YUV as soon as possible and stayed there – H.264 requires it, but I believe we were also usually chroma subsampling to save FPGA resources.
-
Interested in scripting and high-level synthesis thoughts, and in general the question: how should typical ‘user space’ for the end device work? How would you like to design a “patch” for your ideal such device? What kinds of things would you want to explore that aren’t currently possible / are difficult in your current workflow? I feel like there are so many incredible tools for video around and emerging that I need to think on that last one some.
That is probably more than enough babbling out of me! I reckon I need to do some more research on all these different development board options and stuff. Super glad to have found this topic and forum!
3 Likes
Hi! Awesome to have you on the forum! Sounds like you have a ton of experience.
Just a comment on the FPGA hardware thoughts. I’ve been a huge fan of the recent open source yosys/nextpnr/project trellis tool flow (which is why the lattice parts are brought up). One of the downsides to this is that you have no vendor IP, and most hard IP is inaccessible (ie no access to built in I2c controllers).
The way most people make up for that is Litex.
It’s sort of an open-source SoC builder that includes ethernet, DDR3 controllers, HMDI, softcore CPUs etc. You assemble these prebuilt cores using a python script and assign ports and registers so they can connect to whatever you are instantiating in the fabric. (For example the orangecrab board uses this to access its DDR3 ram).
lots of other good ideas and questions in there that I don’t have answers to 
1 Like
Awesome, I was wondering about this. I’m pretty new to the wealth of open source stuff available here, I need to do a lot more reading on this and trying stuff out! Great to see there is a logic analyzer here.
I’ve just found this awesome project which uses an output pin and an LVDS input on the FPGA plus a simple addon circuit to create a sigma-delta ADC for capturing an RGB+sync analog signal. So rad! I have this exact dev board or something really close to it, if I can manage to get an old enough version of the Altera design software to run I may have to dust it off and check this thing out. The github page notes that it has some noise problems and the original source video is pretty low-spec but what a cool proof of concept. Seems like maybe an easy way to get some kind of analog video input by making a hat for one of these open source boards. Also some cool tricks here for improving the DAC resolution.
Just popping in to this conversation. A lot of really good ideas and stuff coming around. To spark the conversation on describing a core system, I guess it’d be good to figure out categories of features, and then start listing and quantifying.
In terms of system access points, it would theoretically have, at a minimum, input channels, output channels, and input modulation, and some forms of feedback, to create something interesting to play with.
I see mentions of LZX’s modules a few times, but haven’t seen Gieskes’ 3trinsRGB+1C mentioned, which also had an interesting feature set.
For signal inputs, I’d want to play with at least two, and be able to perform arithmetic operations composing them with a variety of blend modes. For modulation, it may be nice to access, distort, or replace sync signals. Chromakeying operations would also allow for other explorations.
I’m speaking very much as if this were a fully realized video synthesis piece of equipment, but I think looking at end use cases may improve an early description of the core architecture, in order to realize where we may want to break out control lines, and of what the overall system path would comprise.
I’m just beginning with a homemade video playground, and look forward to seeing how this project develops.
2 Likes
I think there are a couple of interesting possible use cases to explore, which might call for different types of I/O. But one thing I think is great about FPGA platforms is they don’t have to be all things at once, you can use pmods or other kinds of add-on boards to extend the hardware in a modular way, and reconfigure your SoC to only include the logic you actually need for a particular “gateware app”. On the other hand, doing video I/O on a different board can cause difficulties with the bandwidths needed for high resolutions, depending on the interconnect.
Here are some categories I’ve been thinking about, loosely defined and with an understanding that many features make sense in other categories besides the ones they’re listed under. “Prior art” sections are also just scratching the surface of the many years of incredible work many people have put into FPGA tools for video. Mainly just hoping to write some thoughts down and share some research!
Pattern generator / video oscillator
Hardware needs
- RCA genlock input
- I can see most types of output being useful here, depending on what you’re working with. VGA and HDMI output are the most readily accessible - a VGA-to-minijack module or adapter might be the best way to support integrating with a modular system.
- CV input? There are a number of designs floating around for DIN MIDI pmods, or some type of USB control from a computer could also work for “animation” controls.
Prior art
-
Milkymist looks quite awesome. The hardware is no longer available but the group behind this continues to be very involved in open source toolchain development and FPGA hacking – after all “Migen” is short for “Milkymist generator”, see e.g. a framebuffer core here.
- LZX Fortress and Diver modules are prominent programmable-logic based devices and what I think it makes sense to be compatible with, as far as sync connections and voltage levels and stuff, though I personally would be fine with such a thing not actually fitting in a Eurorack case.
-
FPGAWhack is a cool project which lets you compile C expressions to instructions for a pixel processing unit, which I think is a super interesting idea to explore.
- A lot of retrocomputing stuff on FPGAs I think is also a really valuable resource, for example I recently got glitchNES running on this NES implementation, which includes audio and picture processors, using an iCEbreaker board and a VGA pmod, you can imagine adding CV or midi input or other Ming Mecca type features to such a thing. Tons of cool stuff at FPGA Arcade, as one of many examples.
Video processor / effect
Hardware needs
- Analog video I/O
- Multiple outputs, feedback patching considerations. TBC?
- Enough RAM for framebuffers
Prior art
-
LZX Memory Palace is an RGBA framebuffer for patching with LZX video synthesizers.
-
Here is an interesting application converting VGA input into ASCII art on a terminal in real time. (source code)
- The NeTV2 is focused on modifying video content in real time and seems like a very nice platform for digital video – two HDMI inputs, connects to a Raspi. hdmi2usb can be run on this board by using an add-on board to use the PCIe connector for USB. They have a repo for discussing the many applications this kind of platform could have if it were not illegal to circumvent DRM on legally owned video content under the DMCA. The board’s creator is part of an EFF first amendment lawsuit challenging this provision in the DMCA.
- The Open Source Scan Converter is a platform for rescaling older video game consoles to output HDMI to newer displays and equipment. It supports several analog input options, using a TVP7002 video ADC chip.
Capture / utility
Hardware needs
- Act as a USB video device
- HDMI and analog input attachments
- Mass storage attachment
- HDMI preview out
Prior art
-
hdmi2usb is a long running project that’s overlapped with a lot of open source toolchain development. It has a focus on teleconferencing and live event streaming / recording. It seems the board they designed, the Numato Opsis, is no longer available.
- The ULX3S board seems maybe promising here? Looks like there is a GPDI input pmod, an SD card slot, and possibly high speed USB capabilities? At some point? I’m not certain what the status is on that or if it’s achievable with this hardware.
Some general considerations and thoughts
- RAM specs are an important thing to consider for selecting an FPGA board or system-on-module, particularly for framebuffer applications. You can put more memory on expansion boards but it’ll be much slower. As a reference point the LZX Memory Palace uses this SoM which has 512MB DDR3.
- High speed USB also imposes some constraints, historically this has required a USB transciever chip but if I understand right people are starting to do this now with just ESD protection between the USB connector and the FPGA?
- Options for different kinds of expansion boards seem like an important factor for board selection / design since I think there are really a lot of different video I/O requirements for different applications. There is a CSI pmod design that I believe was designed to work with the Raspi camera, perhaps this could also be used with the PiCapture boards?
ULX3S boards are expected to start shipping in another week or so and I’m super excited! It turns out I accidentally ordered 2 of the pmod sets (one each of dual USB, HDMI, and OV7670 camera) – if you got or are getting a ULX3S, DM me and I can send you the spares once they show up! From looking at the repo I believe these have an additional non-standard pair of pins designed for the ULX3S and are not pmods you could necessarily connect to other FPGA boards.
3 Likes
welp i might end up needing the ulx periphs, need to double check on that tho. hoping to start work laying down groundwork for this in november, does anyone have a good “fpga development for dummies” that they recommend for noobs?
Wow sorry I totally meant to respond to this 3 weeks ago. I’ve never really used a book or tutorial series, I get bored really fast and start just coding and squinting at a lot of search result pages for every other error message I encounter. I feel like I had a hard time “getting started” with this open-source toolchain stuff and I could not really describe the steps I took to get a working toolchain but these nmigen installation instructions seem like an OK starting point? For actually programming FPGA designs the best reference for me is example code for whichever specific board you’re using. iCEBreaker has a bunch, so does ULX3S, looks like there are also some for orangcrab.
Historically this stuff always relied on using a ton of clunky vendor software, so Getting Shit Done relied on figuring out how to use some graphical design suite that would probably change its layout every other major version. The open source tooling has improved this situation a lot and once you have stuff installed you can pretty much run one python program that takes care of everything. For verilog or VHDL there are also the language syntax issues but IMO you don’t need to mess with this to get going because the Python nMigen workflow is really well thought out. It’s still useful to be able to read verilog and VHDL because the vast majority of existing code is written in one of them, but at this point I feel like nMigen is the way to go for new projects.
Right now I’m working with the iCEBreaker board and doing a lot of pulling myself into unrelated projects, but I’m gonna maybe try to slowly keep some kind of code journal for this stuff here and roughly sketch what’s going on / how to use it. If you have the Digilent VGA pmod you can currently get this project to show a test pattern and can hold the button on the iCEBreaker board to reveal 64 reference parrots. I could adapt this for other boards that have some way to get VGA output (like the 1bitsquared DVI pmod), when the ULXS board shows up I’ll see where that goes as well.
If you have any trouble getting the software set up or want more detail about any of this let me know!
4 Likes
thanks for the hot tips @csboling ! still havnt found time to dive into this fpga stuff at all yet, but having this info here will really be helpful when i do. will keep an eye on your git-repo too 
1 Like
thanks much! ususally ive found just getting the environment set up for workflows is the trickiest part of this kind of thing, at this point i usually end up taking notes and documenting the process so that i’m never at a point where i have to start over from scratch at any point
2 Likes
lol, i keep trying to get either the ULX3S or orange crab set up for testing about 1nce a month and still have totally no luck. i think what i mean when i say ‘fpga for dummies’ is not some kind of branded book, but literally any kind of getting started with toolchains guide that does not assume any kind of familiarity with the subject
update: after a bunch of research and some failed attempts today, tried on a fresh install of ubuntu 20.04 and found this really really nice .sh for installing EVERY open source toolchain + dependencies that should work with ULX3s ulx3s-toolchain/README.md at master · ulx3s/ulx3s-toolchain · GitHub
it also takes like a half hour to run lol but i’m gonna get at least 1 dang blinky light before this day is done
2 Likes
here is a nightly build of the open source toolchain that I find convenient:
you will want fujprog from the ULX3s repo for programming though.
Still haven’t done any video except for the examples that comes with the ULX3s and Icebreaker but I spun up a little brain board (for the three body) with just the ECP5, Ram, ftdi USB, and 120 or so IO on bergstak lite headers and it seems to work pretty good.
nice, yes i’ve been trying to get the fujprog blinky test thing working for a bit now with no success. keep getting the “Cannot find JTAG cable” error every time. I have a feeling it has something to do with the udev part
Here we have much better success, fujprog is statically linked and doesn’t depend on any other file. Most issues come from user permissions so fujprog should be either run as root or the user should be given permissions to access USB and serial device, that’s usually done easily with some udev rule:
# /etc/udev/rules.d/80-fpga-ulx3s.rules
# this is for usb-serial tty device
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6015", \
MODE="664", GROUP="dialout"
# this is for fujprog libusb access
ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6015", \
GROUP="dialout", MODE="666"
i can see that this file exists over at “/etc/udev/rules.d/80-fpga-ulx3s.rules” but unsure if there is some base level assumption regarding udev rules that i’m missing
one useful command to make sure linux sees the ftdi chip that provides USB for the ULX3S
dmesg | grep FTDI
also just in case, make sure you are using the right USB connector! only one of them goes to the ftdi chip.
if it sees it but FUJPROG is not working it could either be that the ftdi chip is not programmed or you are having issues with the programmer (if so see the ftdi documentation or use this: GitHub - richardeoin/ftx-prog: A Linux command-line alternative to the FTDI FTProg utility for FTDI's FT-X series)
I feel like you may need to link this rules file in your boot script for linux somewhere but I don’t take enough notes and don’t seem to have written it down (if it is even necessary)
there are other programmers: OPENFPGALOADER is one
1 Like
I still haven’t gotten to working on video projects but I recently picked up a couple more dev boards that might be interesting to people here, the tang nano 4k and 9k:
https://gowinsemi.com/en/support/devkits_detail/39
https://gowinsemi.com/en/support/devkits_detail/43/
They feature Gowin littlebee fpgas which is kind of a new chinese brand which has some support in the open source toolchain: GitHub - YosysHQ/apicula: Project Apicula 🐝: bitstream documentation for Gowin FPGAs
Their coolest feature is that you can actually buy them right now but these boards are very cheap and include some support for (and bundles including) cameras and screens.
2 Likes