A pencil case!: Why physical drawing is better in mixed reality

Pencil just released on Quest in Early Access. If you’re passionate about the future of mixed reality for hobbies and education and are curious about pushing the boundaries of 2024 technology, give it a try.

Using a pencil in Pencil

As the writing on the package says and as you will see in the trailer: in Pencil you draw with normal pencils.

Many drawing apps, including the previous Whiteboard in Meta’s Workrooms app, make compromises: Hold the controller backwards and use its butt as a drawing utensil. It’s embarrassing. The pencil rejects the premise: just use a pencil; you already know exactly how that “controller” works. The game doesn’t have to do sophisticated calculations to discern the angle, direction, and pressure of your shot; you do everything yourself. Mastering that “controller” is still up to you.

But you can already draw on paper with a pencil from your living room without $500 headphones, much less a $10-15 app. So why the recommendation?

As of 2024, it is the killer combination of lessons and progressive patterns.

Leo, the Pencil avatar, guides you in real time through a portfolio of drawing techniques. But unlike the lessons you can find on Youtube, Pencil lessons place outlines directly on the canvas. These outlines get more complicated as you progress, and you can turn on their opacity to control your work.

Following a pattern significantly improves my drawing, not only in the quality of the finished work, but also in building my confidence and muscle memory around shapes and techniques. It’s a small physical difference: without an outline, I’d have to glance at and reference my subject and rely on (very, very recent) memory to draw it, instead of drawing what’s directly in front of me. But after just a few lessons I already feel much more confident.

Controls and controllers in Pencil

A discreet MR innovation that becomes a standard: in Pencil, place the left controller on the desk to transmit the position of your paper to the headphones.

Pencil is the latest to take advantage of this trick: Since Meta doesn’t allow third-party access to front-facing cameras, apps will ask you to judge the position of objects in your room via X/Y/Z- proxies of your controllers. coordinates. (Meta’s Horizon Workrooms does the same by asking you to dictate the height of your desk based on the controller’s resting position.)

However, I faced a challenge familiar to anyone who’s ever done stencils: If you move your paper or controller ever so slightly, your outline may be nearly impossible to realign. It makes sense, when you think about it, that the Pencil overlay depicts your paper being taped down.

I kind of complain that Pencil leaves the other controller unused. You use your UI at a distance and in mid-air to toggle your outline settings and transparency, which is tricky. It would be easier and more precise if you could use the buttons and joystick of the unused controller.

Import and export

For creative experiences like Pencil, the Import & Export experience is crucial: sure, it’s nice to draw in OpenBrush, but if it’s annoying to take your creation elsewhere, to the desktop for further refinement, or printed to display on the fridge, or sent to a friend: everyone else gets lost.

Pencil has seamless physical “export” since your work is already in the physical world. “Importing” is another story – inserting models from the Internet into my game was a headache. And given the “little push” issue, I can’t imagine picking up and attempting to complete a drawing based on the WIP outline. Non-contour drawing based on Pencil’s in-app 3D models wouldn’t be a problem to pick up from WIP, but I’m not convinced that’s where the app’s real magic lies.

What should MRI be?

The success of the pencil as an RM drawing aid indicates fantastic potential for the future.

Access to the front-facing cameras seems inevitable and will unlock a superior experience in Pencil. Accurate monitoring would allow me to resume WIP work. Better: Maybe it would give Pencil the power to criticize my work. It comes to mind the old Mario Party minigame “Crazy Cutter”, but my actual drawing skills are evaluated instead of my ability to drive a joystick. Leo should not only give me a final score, but also a real-time score with suggestions on what to correct – my pencil *has* an eraser, after all.

This pencil concept has room to expand beyond its position as a drawing aid. Handwriting comes to mind, and perhaps even practicing arithmetic would be more effective or fun with a character like Leo at the helm. Maybe there’s room for amazing mixed reality games: what if it was a Pencil version of Fruit Ninja have you cut digital objects by sliding the pencil across the paper and the result is something artistic?

We are headed for a future where physical objects – even “dumb” ones, like a regular No. 1 pencil. 2 – they can be brought into digital gaming and where mixed reality can elevate (not just digitize) family activities. Pencil is a first look at what that future might look like. So, for VR and MR enthusiasts: Pencil will spark your imagination and you might even pick up a new skill or hobby along the way.