At Leap Motion, we envision a future where the physical and virtual worlds blend together into a single magical experience. At the heart of this experience is hand tracking, which unlocks interactions uniquely suited to virtual and augmented reality. To explore the boundaries of interactive design in AR, we created Project North Star, which drove us to push beyond the limitations of existing systems.
Today, we’re excited to share the open source schematics of the North Star headset, along with a short guide on how to build one. By open sourcing the design and putting it into the hands of the hacker community, we hope to accelerate experimentation and discussion around what augmented reality can be. You can download the package from our website or dig into the project on GitHub, where it’s been published under an Apache license.
Our goal is for the reference design to be accessible and inexpensive to build, using off-the-shelf components and 3D-printed parts. At the same time, these are still early days and we’re looking forward to your feedback on this initial release. The mechanical parts and most of the software are ready for primetime, while other areas are less developed. The reflectors and display driver board are custom-made and expensive to produce in single units, but become cost-effective at scale. We’re also exploring how the custom components might be made more accessible to everyone.
Project North Star points a future where the physical and virtual worlds blend together into a single magical experience. Click To TweetThe headset features two 120 fps, 1600×1440 displays with a field of view covering over a hundred degrees combined. While the classic Leap Motion Controller’s FOV is significantly beyond existing AR headsets such as Microsoft Hololens and Magic Leap One, it felt limiting on the North Star headset. As a result, we used our next-generation ultra-wide tracking module. These new modules are already being embedded directly into upcoming VR headsets, with AR on the horizon.
Project North Star is very much a work in progress. Over the coming weeks, we’ll continue to post updates to the core release package. Let us know what you think in the comments and forum thread. If your company is interested in bringing North Star to the world, email us at partnerships@leapmotion.com.
It’s time to look beyond platforms and form factors, to the core user experience that makes augmented reality the next great computing medium. Let’s build it together.
Anyone can build the headset, but nobody can buy the tracking module? Smart. You guys have been sitting on Dragonfly for years. What’s the idea?
June 6, 2018 at 5:58 amThe design can be easily modified to accommodate a Leap Motion Controller, and the only benefit of the newer modules is the wider field of view, which is not strictly necessary for interaction prototyping on a reference design. In any event we do occasionally make the newer modules available to some developers (e.g. you would have seen our tech at SXSW and the Tribeca Film Festival).
As for Dragonfly, we stepped away from its development because we felt that passthrough AR was not the right path, particularly not for us at that time. Sharing our internal work is a double-edged sword, as there’s a wide gulf between “here’s a cool thing we made” and “it’s available on store shelves now.” Some things become products, others don’t.
June 6, 2018 at 6:14 amYou need to continue to miniaturize the module, figure out how to do the skeletal computing with minimal latency onzboard, and simply export the real time tracking data. Make it an easily interfaceble module with Android, Arduino, etc and I strongly suspect you will have much much wider uptake for your gear and vision.
Also when the hell is an AR/VR hmd with your new embedded hardware coming out? All this gear keeps coming out with crap controllers and yet nothing from you guys.
And even when headsets do come out featuring your hardware I am sure they will come with all the usual software bloat of the companies you partner with as they try to lock users in to their shit software gardens.
Something isn’t right here. Make a kickstarter to fund a suitably large production run of your northstar headsets using your latest wide FOV sensors, keep the software and hardware platform open and you might be surprised…
Punting to open source community, incompletely at that, comes across as lazy. You guys are gonna get surpassed by a competitor with a more complete vision.
June 7, 2018 at 4:50 pmYou need to do a kickstarter so we can get some cost effective kits together!
June 6, 2018 at 6:18 amYes finally I’ve been waiting so long for this! Took a quick peek at the build guide and I have a couple of questions. First do you guys plan to bring out the hand tracking module/video driver? If so any estimate when? Second do you have any suggestions where I could go to make these eliptical reflectors? I’ve tried DIY’ing them a while ago calculating the dimensions on my own but actually producing them is pretty hard to do at home. Thank you guys for open sourcing this project, it’s truly amazing! Next school year I’ve got to start on my final work for highschool and since this fits perfectly in the area I’m studying (industrial engineering), I’m actually really considering building this and improving it (mostly structural / building software)
June 6, 2018 at 6:26 amAlso went looking for the display panels and could it be there is a typo or are these not available yet? The guide says the model is VS035ZSM-NW0 – 3.5inch LTPS 1440*1600 Module
When searching on all types of display sites I only came across VS035ZSM-NH0 modules which have the same resolution/size and can supposably get to these refresh rates:
120Hz (60HZ Input) : Embedded MEMC Circuit
120Hz+120Hz : 120Hz Panel+120Hz Backlight
240Hz (60HZ Input) : Embedded MEMC Circuit
It’s also the same manufacturer BOE so did I find the right display? Don’t want to risk buying the wrong ones and coming to the conclusion that they dont work with the display driver.
June 6, 2018 at 6:50 amThe VS035ZSM-NH0 model seems to be the 60hz from what ive seen on this site. http://www.panelook.com/VS035ZSM-NH0-69P0_BOE_3.5_LCM_overview_28892.html
I’m not really sure where to find the NW model where the refresh rate is 120hz though. It seems like getting all of these parts is quite difficult.
June 6, 2018 at 12:18 pmYeah got to the same site as you Tomgie. If you hover over the refresh rate spec then it shows the refresh rates I mentioned here above. After some googling I found this forum post: http://www.avsforum.com/forum/166-lcd-flat-panel-displays/1458177-tv-panel-refresh-rate-confusion-help.html
The way I understand it is that the image is 60fps but is refreshes at 120/240Hz to prevent flickering issues. Could be that I’m wrong but seems the most logical to me. Seems like we’re out of luck on this one. You guy’s should really put something together where people can aquire these parts a lot easier because I don’t think a lot of people will be able to build this.
June 7, 2018 at 12:57 pmYou definitely do not want a panel that refreshes faster than the number of whole new images that it can display. For VR/AR use cases, that behavior will result in really bad ghosting artifacts.
That general model of panel can support at least 90Hz actual refresh rate (and “global display” behavior, where the backlight is pulsed once per frame after the panel updates – really well suited to VR/AR), though I don’t recall the exact models I’ve used. It is important to note that it’s specific setup commands, etc. that controls this on capable panels, it’s not enough just to get a capable panel and run it with generic init routines. I know they’ve used a few different layouts and panel connectors on related panels, which might be the difference in those model number suffixes. (Or it may be that nobody leaked data sheets for the specific variants as used in VR, since that’s typically where those panel comparison sites get data if I understand correctly.)
They are nice panels if you can get them and get them configured for VR correctly, though! Bright and very crisp looking.
June 7, 2018 at 7:03 pmI hope they set something up to buy parts trough them, they could easly make some extra cash. I don’t want to buy like 20 or even a hundred displays just so that I can use 2 of them. I’m also pretty sure that the majority of people wouldn’t mind paying a little extra for them if they can get them individualy
June 8, 2018 at 3:35 amBOE does produce 3.5”1440*1600 modules and calls them VS035ZSM-NW0, though you won’t find them publicly listed because they only sell them in large amounts (e.g. a hundred units or so). We’ve gotten an extraordinary amount of interest in North Star so we’re definitely exploring some options as far as making parts more easily available.
June 8, 2018 at 6:16 amAs someone pointed out starting a kickstarter and people buy it basically there would be wonderful. I would buy a like 6 of them if I could.
June 14, 2018 at 2:46 amUnfortunately VS035ZSM-NH0 is no longer produced. BOE does produce 3.5”1440*1600 modules and calls them VS035ZSM-NW0, though you won’t find them publicly listed because they only sell them in large amounts (e.g. a hundred units or so).
June 7, 2018 at 10:02 amIf you don’t have the PCB creation experience or SMD soldering & mounting tools, there is the OEM HDMI-MIPI board you can buy on Alibaba that I have used for another project that you can get programmed for the two panels shown here https://www.alibaba.com/product-detail/Confu-HDMI-to-MIPI-DSI-Board_60748904965.html?spm=a2700.7724838.2017115.1.4b98352cTrEBfD. If you want to use a OLED panel and don’t mind the resolution being less, you might look at the Topfoison 3.8″ OLED 1200×1080 (Rift CV & Vive) (http://www.topfoison.com/TF38101A-dual-oled-screen.html) whom I have worked with using their single 2560×1440 6″ panel that also includes a slim-line HMDI-MIPI interface board similar to the one Leap Motion has created and provided a schematic for. Since Leap Motion board does not include a IMU in their display interface, I am assuming that it is included in their tracking board but you could easily turn this into a VR headset with a number of IMU kits that are SteamVR compatible. I will talk more about alternatives this Friday.
June 6, 2018 at 1:40 pmNote that topfoison drivers sold widely won’t necessarily set up a given panel for ideal low persistence, high actual frame rate mode as needed for VR/ar. (They might, especially if you get in touch with them for something specific, but by default I wouldn’t assume anything since they market stuff for a wide target audience.) From just a quick glance at the schematics (no pcb layout though? These are high bandwidth, “picky” signals so not a great “my first pcb layout” candidate…) it looks like Leap Motion here has released a nice display controller board, and of a newer generation than the 3.81 1200×1080 panels you mention: this is a very fast and capable bridge/controller chip used in several Windows MR headsets, which provides roughly double (or more) the performance/capability of the (earlier) controller/bridge used in cv1/Vive/hdk2 and that mentioned topfoison kit (display port and 2x2x4 mipi lanes, possibly at a higher rate even iirc, vs HDMI 1.x and 2×4 mipi lanes at 1gbps). And, these panels require that, both because of the higher resolution and because they’re global display capable but require additional vblank to do that in. I personally wouldn’t assume that a “generic” display controller board can drive these panels and their demanding backlights without first asking the vendor, ideally with VR-focused data sheet in hand. They’re pretty advanced, special panels – not really the kind of thing like many 5 to 6 inch panels where they’re mainly a smart phone display first, and may be well suited or poorly suited to VR. I don’t know what else besides immersive displays would use this size and resolution of panel with such nifty VR features. (On paper these look similar to the ones in the Vive pro, though the underlying tech is quite different)
June 7, 2018 at 7:33 pmThe Toshiba TC358870XBG is capable and is the one used in both the CV1 and HTC/Vive. If I remember correctly the Topfoison’s board also uses this and is capable of driving the panels at 90Hz. Of course it is not 120Hz, but as you have mentioned unless you are pretty good at creating high-speed PCB, just creating something from the schematic could be frustrating. I also agree on the generic display driver on Alibaba may not be set for 120 Hz, but I do know it is capable of at least 60 Hz. These are just options to those who want to experiment and not looking for the perfect solution, that even the Magic Leap project will not be perfect if you can round up all the parts to complete it.
Which leads me to something I have been working. With the cost of the WMRs now about $225, and with its 3″ panels and stereo SLAM, this could be great solution if you salvage the parts. You won’t be able to push the panels out to the sides, but there are other options that give good results. Still working on it, so nothing to show yet.
June 8, 2018 at 2:21 pmYes, the chip I was thinking of is the tc358870. From experience, you can’t run these panels in VR more well using it: you can barely run them at 90Hz with one chip/HDMI link per panel (it’s very finicky because you have to both overclock the bridge chip and exceed the HDMI spec – so not every unit behaves well or reliably, and there is GPU incompatibility.) The pixel clock is just too high (up around 330MHz when spec is 297).
The other thing about a generic driver board is that it probably won’t have the special init sequence or timings to enable “low persistence” or “global display” mode. These require different video timings (on the boe panels at least) and different init commands (on both the boe and the 1200×1080 panel kit, which I’ve also used) from “generic usage”, so if you aren’t specifically configured for those modes (with specially chosen backlight drivers, in some cases), you’ll still see an image, but it will be higher persistence, possibly flashed twice for each frame (bad!), and probably incremental display so you might see artifacts from scanout if your software stack doesn’t do per-scanline-warping.
It would be very interesting if someone could get a wmr device and write open source replacement firmware for it, to allow the panels and display controller/bridge to be reused separately. If anybody wants to fund it, that sort of reverse engineering is within my abilities for sure! 🙂
June 12, 2018 at 8:45 amI knew your name sounded familiar. Do you still work for OSVR? We left messages back and forth on Github as I was trying to find out more about your Android Libraries. With that being said, I really respect your knowledge and the work you have done.
I also went back to see if I had remembered correctly if the tc358870 was being used in the Oculus CV, and it was (https://www.ifixit.com/Teardown/Oculus+Rift+CV1+Teardown/60612) . And though I agree with you with your assessment of it being pushed to its limit, Toshiba does have another version that is faster. I also agree with your assessment that most generic boards are not tuned for low-persistence, but I don’t know if that is as well suited for projected image AR. In my experience of projecting a Pixel XL in Daydream mode, the quality of the projection is more translucent. I would think you would have to create a new mode that has a high refresh rate, and high image brightness. at 120 Hz may lend itself better to this, even with frame doubling. Also, since your background is black when no object is being projected, an actual hardware blanking would be a better solution than just submitting a black pixel.
As far as the WMR system is concerned it uses a Cypress CYUSB3064-BZXC as its MIPI controller. Do you have experience with this chip? I have done some work already removing the board and cameras, (even cut out the plastic camera support out and mounted the cameras back on it to ensure proper spacing and angle) and even though IMU based head rotation works great the inside/out tracking is unstable. There is a lot of copper shielding tape in the HP HMD surrounding these cameras so I am thinking possibly noise could be the issue, but I have since wrapped the flexible PCBs with copper tape with no improvement. I am hoping to get more design information on their camera positioning and hopefully a test app to verify camera output. If I hold the controller up in front of me, it seems to track the controller pretty good but still has some glitches. My actual movement through positional tracking is extremely unstable though. I also removed the clear plastic plate that covered the cameras and sadly threw part of it away. Now I am thinking this could have actually been a filter instead of just clear plastic to protect the cameras. In any case a little disappointed at this point.
June 12, 2018 at 1:02 pmhttp://www.reddit.com/r/ProjectNorthStar for anyone who wants help building one!
June 6, 2018 at 5:19 pmThanks, this looks pretty swell! You might want to clarify the mention of the licensing: it looks like at least the firmware is actually GPL3, not Apache 2. (And I’m not entirely sure how Apache 2 works when applied to hardware designs, but the common and permissive license choice is nice at least in theory, and it sends the right message.)
June 7, 2018 at 7:50 pm