Shoppe Keepers and Vehicles Update! November 2020

The previous update which focused on engine optimization and the introduction of the Blacksmith shoppe keeper was back in June. Life has threw me some curveballs since but I have finally managed to complete all remaining shoppe keepers and introduced the common vehicles including Magic Carpets and Ships.

Shoppe Keepers

These shoppe keepers are comprised of:

• Bar Keepers
• Inn Keepers
• Guild Masters
• Healers
• Apothecaries
• Horse Sellers
• Shipwrights
• Blacksmiths (already implemented)

The shoppe keepers all use a common interface originally seen in the Blacksmith update. There is an emphasis on retaining the original charm from the smack talking shoppe keepers while improving and streamlining the shoppe experience. For example, it is VERY easy to miss the gossip option in Taverns if you are just quickly picking up rations and leaving. In the updated Bar Keeper interface, the gossip option is easily seen and will save many players hours of frustration by giving hints for a small fee.

I also opted to allow you to try to buy items and services if you don’t have the money. Most modern games would simply grey it out – but I just love the smarmy responses (or violent response if you try to stiff the Inn Keeper!).

Vehicles

The classes Ultima V vehicles were added as well. This was a hard requirement since you will visit the Shipwright and Horse Seller and expect to actually receive the thing you bought. The vehicles include:

• Frigates
• Skiffs
• Magic Carpets
• Horses

The Details

This update was very grueling! It seemed that everytime I had to introduce the next shoppe keeper a new challenge would present itself. For example, when implementing the Shipwright and Horse Seller I had to completely overhaul the underlying sprite handling system. I scoured the previous reverse engineering of the save state to understand how to determine if the party had boarded a vehicle. It wasn’t fully documented, but as always the previous researchers provided invaluable hints. The super cool news is that the same sprite systems that are used for riding magic carpets etc is also used to display “odd” sprites such as the bats in the tower in Skara Brae – something that had been on my list for a while.

It was also an opportunity to begin to describe the monsters as they would appear in the overworld.

It has lead to me some interesting bonus feature ideas such as a travelling merchants in the overworld (there are no NPCs in the overworld currently). Just ideas for now, but lots of cool opportunities in the future.

Continuing Challenges

I did what no one is supposed to do, I chose a very large project as my first Unity/Game project. As a result I have been “learning on the job” and creating some pretty crummy early interfaces and code. You will see UI improvements with the shoppe keeper update such as the “player selector”. There continues to be a lot of previous work such as the ‘U’se item UI which is a bit rough around the edges and doesn’t even support mouse input. It’s not really a problem, just notable.

Around The Corner

I have had a few people ask for up to date demos. In the next few days I will release Windows, MacOS and Linux pre-alpha builds for testing and experimenting.

What’s Next?

As mentioned in the video, I will:

• Spit and polish the UI and graphics
• Add line of sight (attempt #3)
• Either “monster stuff” or dungeons

The spit and polish involves updating some old UI elements to be keyboard+mouse capable and just provide a more consistent overall experience. I also believe there is a long way to go on the graphical fidelity and tuning. I have a long list that I maintain of things I could do better that I hope to tackle.

I will AGAIN try to add line of sight – it just never makes it high on my list and I am seriously worried about screwing it up – but it’s long overdue.

Lastly, I am desperately keen to move onto some “very different” from the stuff I’ve been working on so I hope to begin to tackle either some combat elements or dungeons. I will just see what side of the bed I wake up on.

June 2020 Update – Engine Update and Blacksmiths

This update has been about 3 months in the works. You would think with Covid and self isolation that I would find myself with more free time, but it so often seems like I have less. Either way, there were two major features that I chipped away at over these three months and have made a big impact to the game.

June 2020 Update Video

Engine Update

Although I have written to graphical components in Unity and Unity is in fact a game engine, I had a tough problem to solve – the world is round. Unity is built primarily for a world that does not repeat. In Unity, when you hit the end of the defined worldspace then it just ends.

My first implementation of the worldspace involved keeping the Avatar at 0,0 in the world, while literally moving the visible portions of the world directly underneath the Avatar. This solved an important problem, but introduced a huge expense to the CPU and created unnecessary lag between input and screen response to that input. In short – it was lazy, and I paid for it in performance.

The new version of the engine is built on a different principle. The Avatar actually moves through the 3D space, while the world stays put. There are some optimizations such as trying to only draw the visible space so I don’ have to load the entire world in memory as I walk around a smaller portion of it. This resulted in some pretty crazy math-fu, but in the end it pretty much doubled the framerate and took a huge burden off the CPU. The input lag is gone and I am finally happy with the play experience.

Original Framerate Sample

New Engine Framerate Sample

A new Dev Vlog will be appearing shortly in which I will walk through the two models, the advantages, disadvantages and some of the optimizations that were implemented to squeeze every last frame out.

Shoppe Keepers and the Blacksmith

There were two main improvements I wanted to make to the blacksmith:

1. Provide more equipment stats to help make buying decisions easier
2. Incorporate the original (often funny) dialogue into a point and click interface

When I set out to implement the many shoppe keeper varieties, I thought it was going to be a snap. Track down the conversation data in the “talk” files, make some buttons and we would be done before you know it. Unfortunately the reality of the shoppe keeper data and code immediately reared its head.

In short – the shoppe keeper conversation data is spread across multiple files, if it’s findable at all. Next thing I know a whole new shoppe keeper dialogue system needs to be created, pulling all that disparate data from it’s many locations.

Although I only got the blacksmith implemented, I was able to create a complete shoppe keeper UI system which makes it much easier to use the same framework to add the additional shoppe keepers.

What’s Next?

Well, for the second time in a row I intend on implementing line of sight ensuring the player only sees what they are meant to see (turn off the x-ray vision!)

Secondly I am going to continue to implement the remaining shoppe keepers who include:

• Barkeep
• Horse seller
• Shipwright
• Healer
• Innkeeper
• Magic Seller
• Guild Master

I am sure I will drop in one or two distractions as well – spending too much time on one feature can stifle motivation and creativity IMHO.

First Dev Vlog

Well, there isn’t much to say except that I have finally took the plunge and posted a Dev Vlog. This first vlog will introduce the tools I use to develop and offer a few antidotes while I’m at it. I talk about game engines, IDEs, art tools, editors and stuff.

In the future I will deep dive into particular subjects for those who are interested as well. In fact, I would love to hear about any particular subjects you may be interested in, in the future!

About Time! April 2020 Update

So What’s New?

I have already posted about NPC AI/Pathfinding, debug console and the Pre-Alpha Build, so I will stick to the newer items.

• Full day/night cycles
• Full moon cycles with onscreen widget
• Moongates and Moonstones
  • Moongate travel by moon cycle
  • Cool “up/down” animation of moongate
  • Cool fade in/fade out when teleporting
  • Ambient glow of Moongates
  • Search and ‘G’et the moonstone
  • Bury the moonstone
• ‘P’ush mechanic
  • Did you know chairs change direction when you push them? Now I do!
• Complete Z-Stat descriptions
  • Highlighting of Reagents and Spell names for easier and quicker reference
  • Thank you to Ultima Codex who is attributed to many entries that were not in the original Book Of Lore
  • Look out for some great entries from the Book or Brad
• Small enhancements include:
  • Fall down grates (even multiple grates like in Blackthorne’s castle)
  • Begun using light emitting materials in Unity to make things glow
    • Magic doors
    • Lava
    • Crystal balls

Some Details and Challenges

This was an interesting update because it seemed every time I wanted to implement some small component of the feature list then I would end up having to create some new system I had never considered.

Picking up a moonstone

So you want to ‘G’et a moonstone? Well first you ‘S’earch for it of course. But when you search in game for things that aren’t moonstones you could have stacks. The stacks do not translate well into the 3D game since they stack. So I had to create a queuing system that essentially puts them in a first in first out (FIFO) structure where the next item would appear as you ‘G’et the “exposed” item.

Next, typical inventory items are stored in a consistent way in the saved.gam file which makes it super easy to keep track – except moonstones are different. Moonstones are either “buried” or “not buried” and if they are not buried then I must assume they are in the Avatar’s inventory. This does not quite fit with the original games logic, and I still have to solve for it (but as I write this, I think I figured it out 🙂 )

Moonphases

Frankly, I have gotten away with being fairly ignorant of exactly how the moonphases work, progress and why the hell there are two moons! Needless to say, there is some very dirty (yet functional!) code that abstracts those complexities. I even wrote (borrowed) some calculus to move the moons and Sun in a circle. May God have mercy on my soul if I need to update this code again!

3D Tiles in a 2D World

This is not a new problem to me – the original game only showed a single tile at a time, so if an item such as a moonstone was exposed, then it simply hid the tile underneath. Also, things like signs did not technically have a tile underneath.

I had implemented a guessing algorithm in the past to try and take a best guess at the tile underneath but it was often wrong (no-one’s perfect), so to combat that I implemented an unfortunate, but requisite tile override system that allows me override any tile on any map. Note: This could be used in the future for customization or to makes quality of life tweaks to the game (I’m look at you @Cambragol!).

The Core Library versus the Unity Game Code

As many are aware, this is technically two projects – the core game code called Ultima5Redux (soon to be UltimaVRedux) and Ultima5Redux3D. Because I am using a paid/proprietary module in Ultima5Redux3D, I am unable to share it publicly on GitHub. To make sure the community get’s as much value as possible, I have continued to move game logic to the core library – if done correctly, the Ultima5Redux3D will only be reacting core game libraries logic decisions, allowing anyone in the future to build ontop of the core library if they opted to (MIT license).

Thanks

Just wanted to say thank you to all those who actively participate in the project such as the great feedback and ideas from @Cambragol, the publicity from Dino and WTF Dragon and all the other dragons who give me encouragement on Twitter and Reddit. It is always appreciated!

Also thank you to the Ultima Codex wiki contributors! When filling out the Z-Stats descriptions I used the wiki exclusively for quick descriptions from the Book of Lore, and when they weren’t available then I quoted (and attributed) portions of an article directly.

Let’s Walk on the Wild Side!

Coles Notes – there is a pre-alpha demo for you to try!

There is a pre-pre-alpha demo available for Windows and MacOS here: https://github.com/bradhannah/Ultima5Redux-Demos/blob/master/README.md! This version is not a game – it some core mechanics with endless bugs, but it’s something 🙂 I hope you like it. Please follow the instructions, if you don’t then you will only be disappointed.

So what’s new?

Since last update I have added a number of features:

• Improved pathfinding
  • tile weighting allowing NPCs to favour dirt paths and brick roads over grassy shortcuts
  • smarter wandering and collision correction
  • comprehensive simulation of NPC travel over full day for every single town, keep and every single NPC
• Improved directory/save game selection menu which was previously very buggy
• Klimbing down grates and falling through floors (think Blackthorne’s Palace!)
• Mirrors now reflect and can be broken by Avatar
• Fixed a ton of long standing bugs
  • NPCs getting stuck/deadlocked
  • “Bleeding tiles” on outside tiles of townes
  • Conversation bugs causing crashing with gold or item transactions
• Upgraded to latest High Defintion Render Pipeline for Unity
• Created first pre-alpha release
• Performance improvements and optimizations (was at 2fps in Blackthorne’s castle before I did!)
• Added Unit Testing framework (NUnit), writing some initial tests
• Initially migrated all tasks to Trello and then GitHub Projects
• Updated MindMap

This work is now all being tracked here if you want to follow along https://github.com/users/bradhannah/projects/1.

Interestingly the biggest chunk of time was spent tuning the pathfinding, taking it from “good enough” to “not too shabby!”. I was able to write a number of simulations which both exposed a TON of deadlocks, but also let me address them, solve them and then prove they were solved.

What’s Next?

Well if my project list is any indication then I will intend on focusing on the following (as always – subject to change):

• Gradual sunrise and sunsets
• Creating a tile override system that will allow overrides for “poorly guessed” floor tiles
• ‘P’ush things around the maps
• ‘T’alk to shopkeepers
• Moongates!!
• Rewrite of voxel caching system to prevent “double caching” annoyance
• Complete population of remaining item and inventory descriptions for Z-Stats
• Fix some “special NPCs” that do not have proper sprites
• Add “conversation log” feature for recording NPC conversations

Mindmap Update

Also here is an updated and already out of date mindmap for those who like pretty pictures.

Pathfinding – The All Consuming Chore

NPC AI and Pathfinding

As you may be aware since my late December update I have been focused on making the NPCs more mobile. I had been dreading the work because I feared (and was correct!) that it would be disruptive and force me to refactor my code in a big way.

At it’s most basic, every NPC has a schedule which basically has four main components:

• What time should I move to a new position?
• What X,Y coordinate should I be homed at?
• Which Floor should I homed at?
• When I get there, how should I act?

So we have interesting questions to answer like, if they aren’t on the right floor, then which ladder or staircase do they use? What do you do if two NPCs are battling for a single tile? Mix all that in with the memory and CPU constrained implementations developed by the original developers and you have a bit of a mess.

We also had the problem of reading in save game information that states where an NPC is presently and if they already had a path planned. If they did then we need to read it in and follow the original plan. One of my main goals was to have as close to 100% compatibility with the original save state as possible. To the point – ideally – where you could bounce between the original and Redux freely. That may be a pipe dream, but it is still my intention. Following this principal will also ease playtesting because I will be able to do like for like playtesting.

Code Improvements

To accommodate the additional logic and to make my life much easier I spent the better part of the month implementing a new concept called the Virtual Map (https://github.com/bradhannah/Ultima5Redux/blob/master/Ultima5Redux/Maps/VirtualMap.cs) It essentially pulls in all factors that could or would affect the map such as NPCs (dead or alive!?), open doors, keys in trees etc and give the program a single source of truth.

Secondly I fleshed out another concept called the World (https://github.com/bradhannah/Ultima5Redux/blob/master/Ultima5Redux/World.cs). As you may be aware – the project is broken into two parts; the core library Ultima5Redux and the 3D game component written for Unity called Ultima5Redux3D. The creation of the World object actually moves nearly all game logic into the core library, meaning that anyone in the future could “easily” plugin the core library and build on top of it with all the benefits of state management, world interactions, and even automated AI (crude) handling. It may seem a bit of a bore, but this was a massive shift for the project which ultimately is meant as an asset for the amazing Ultima Dragons community.

Another problem I had to solve was pathfinding. I have cringed at the thought of implementing my own pathfinding (bad college experience!). I knew everyone used the A* algorithm, so I went searching for a solution that was already baked and free OSS. I found https://github.com/davecusatis/A-Star-Sharp which is a wonderful and very C# native implementation. A small bug fix later, and the new Virtual Map function and I was up and running.

Lastly, voxels are 3D objects, and with that come with quite an expense. I managed to get around to something that has been on my list for the last 6 months. A simple voxel caching system was implemented which saved the unnecessary loading and destruction of voxels as the Avatar moved around the world and cities. It uses only basic constructs like Dictionaries and Queues to save an reuse tiles whenever possible. It more than double the framerate during my testing. As you can tell below, it had some hiccups!

Debug Console

I struggled with another issue – the debugger and debug console have become increasingly sluggish as I moved more and more into the core library. Secondly, it was very disruptive to continually break and inspect so I added a debug console which can be toggled as needed. As a bonus for any C# nerds, it is connected to the Debug/Trace handlers so all System.Diagnostic.Debug.WriteLine outputs automatically stream to it.

What’s Next?

The next month will be spent paying down some technical debt that I have accumulated as well as perhaps some core updates.

• Tidy up and debug small bugs with pathfinding and scheduling
• Improved lighting transitions from day to night to day
• Save some state to disk such as player position and movement directions
• Populate remaining item descriptions for ZStats
• Continued play testing of the original game to refamiliarize myself with the finer details (F* this game is hard)

Maybe more – but given how slow it felt like I moved last month, I will hold it there.

Holiday Update!

I have posted a new video highlighting some features I have shown on Twitter (ready/use) and some I haven’t (ZStats).

The focus of the update is the following:

• Intro screen with FIRE animation
• Implemented “U”se interface
• Implemented “R”eady
  equipment interface
• Implemented “N”ew Order
• Implemented ZStats
  • Character Stats
  • Item Descriptions

There a number of work items that are far less visible as well. A lot of time has been spent learning the ins and outs of the Unity UI. I actually implemented the few new UI features with varying techniques depending on time of day, phase of moon and general mood 🙂

Also, I implemented the Ready/Use system first which emulates the original experience fairly close, focusing on keyboard input and a general text based interface. The ZStats was far more graphical and extended existing functionality.

I will undoubtedly spend the month wondering how or if I should re-implement the Ready/Use interface to match the aesthetic of the ZStats interface.

What’s Coming

Next up is a focus on more fundamental mechanics that actually make the game feel more polished and “more like a game”.

• Finish adding sprites and descriptions for
  remaining ZStat items
• NPC pathfinding and schedules
• Time of day/lighting change
• Save game progressions to disk
• time of day
• equipment/inventory changes
• (Maybe) some basic non-combat spells

Graphics – they’re hard

With the completion of the “R”eady and “N”ew Order functions, I have opted to mix it up and move on to something quite different. I am actively working towards getting an intro screen up and running. The primary goals will be:

1. Show wicked blue/greeny Ultima logo
2. Show Warriors of Destiny animated fire logo
3. Provide load function – selecting from dynamic pick list of save directories

Other functions will follow eventually, but the biggest pain in the ass I currently suffer if manually changing my save game directory in code every time I want to test a new scenario (ie. new equipment load outs, different party members).

Progress thus far

I had in the past experimented and loosely integrated the Ultima 6 Decoder (http://nodling.nullneuron.net/ultima/code/ultima6/u6decode.zip). This uses an LZW algorithm to explode the *.16 files. The trick is then taking these exploded files and exporting them one more time – to a more commonly read format such as bitmap (.bmp).

When opening the create.16 file, I worked on spotting patterns and managed to spot what I incorrectly identified as fire.

To make sense of the specific format of the file, I looked back in time at the Ultima 4 ( http://wiki.ultimacodex.com/wiki/Ultima_IV_Internal_Formats#EGA_Files ) details. From there I created a basic program to read in and create a bitmap (https://github.com/bradhannah/Ultima5Redux/tree/master/Raw16ToBMP ). This is when i found out that my “flames” were actually the fortune teller.

Realizing I picked the wrong graphic I moved on to startsc.16.

So far so good. I have discovered a few things:

1. each pixel is encoded in 4 bit chunks
2. there are non-pixel bytes of varying sizes before each graphic which i haven’t been able to identify yet

This is what I’ve got so far – but my ultimate hope is to dynamically load the graphics at runtime from the original data files instead of exporting to BMP. I want to make sure that I am not peddling in copyrighted materials after-all 🙂

November 2019 Update Video

Novembers video includes some great updates. The project has really begun take shape and is actually starting to look like a game.

Note: All character data, inventory data and string data is gathered directly from actual Ultima save and data files

A few key changes/improvements in the November update

Visuals

• Shifted to a classic isometric angle
• Completed remaining sprite conversions to 3D model equivalents
• Upgraded lighting engine

HUD

• Implemented HUD with player characters, basic item quantities and current location (also added time into HUD!)
• Implemented new Location tracking in HUD

NPC Chat

• Reimplemented UI for NPC conversations with common look and feel

Signs

• Implemented signs (no runes yet!)

Inventory System

• Implemented basic inventory system

Basic Function

• K-limb fences and mountains
• O-pening doors
• J-immy’ing doors
• Space – pass time
• G-et torches from wall
• Porcullis drop at night

Next month brings on some new challenges I think. I will continue to work on equipping weapons and armour… you know honestly I’m not sure what else I will do – I just kind of work on the thing that most interests me that day or if I’m short on time, I just fix a bug and move on.

Signs Signs Everywhere are Signs

I haven’t found myself with much free time to work on my project in the last few weeks but I did manage to get some basic signs up and running.

It’s actually quite interesting the way that signs are stored. At least 2/3rds of them. There is a signs.dat file that contains the majority – however for some unknown reason I have unable to find a file with all the Eight Laws signs. As a result I have had to hard code them for now.

Also many signs use non english characters which I believe are indexes into a font file – but I haven’t got to the point where I can analyze them quite yet.

Lastly some signs are stuffed in the data.ovl file. I can only imagine why they needed to sink signs into any file other than signs.dat. I haven’t had to hardcode the sign text, but have had to hard code some file offsets which is a little gross, but I’ve done it before.

Thank goodness for the original rev-eng work here: http://wiki.ultimacodex.com/wiki/Ultima_V_Internal_Formats#SIGNS.DAT

Also I have begun to update some of the finer details in the wiki starting with Signs and NPCs.

Mind Mapping to Get Organized!

I often use Mind Maps at my day job to help organize my otherwise chaotic brain! I have begun to map out the entirety of what Ultima 5 boils down to in terms of function – then I overlay my personal plans for how I want to improve it. For example I have defined a number of custom UIs for functions such as buying/selling and a conversation log that you can actually refer to well after you had the conversation!

More than anything, it makes me realize how far I really have to go! But, it will serve as a de-facto project plan and feature list.

• 

October Update

Take a look at the latest video!!

Work continues slowly but surely on the project. A focus on improved visuals has gone well (but is far from complete).

Updates include:

• more 3D voxels replacing legacy sprites
• ambient lighting
• twilight lighting
• new daytime lighting
• basic NPC dialog

For the Unity nerds out there – I have switched from built-in renderer, to light weight render pipeline (LWRP) and finally to high definition render pipeline (HDRP). This has been a really neat experience and continues to strengthen the visual quality.

Next up I expect to:

• Complete all “standard” NPC schedules and textures
• Actually open a door!
• Start creating a permanent UI

The cat’s out of the bag

Well I have been developing since the beginning of July and quietly uploading to GitHub. I decided to actually tell/show people what I have been up to.

You can read some interesting Q&A on Reddit here: https://www.reddit.com/r/Ultima/comments/d2z1iu/ultima5redux_a_very_very_early_preview_i_have/

Or you can just watch the development update video here:

There will be more to come but this seems to be a good start.