Nearly a year ago I blogged my review of "No Man's Sky", a game that garnered near universal hate or love, with not many people in the middle. Since that time, the game has been updated a few times and it has become incredibly rich. I'm now on my second play and I see myself dedicating all of my game playing time on it for the next few months. (This is also the first game I bought for both console and PC.) For a good look at what's changed, I recommend Kotaku's article: "No Man's Sky Is Good Now".
One of the biggest aspects of the game is crafting, and while that can sometimes be a chore, it can also be satisfying to build out your ship and base. I noticed that I had a hard time keeping track of what I needed to gather. What I mean is - consider that I want to build item X. It needs 50 iron and 50 platinum. That's easy enough to track, but consider item Y. It needs 50 iron and platinum too, but also a DooHicky. A DooHick needs 25 iron and 100 beers. (Ok, I'm kinda making this up as a go.) So the total needs for item Y are 75 iron, 50 platinum, and 100 beers. It gets more confusing when you add multiple items you need.
I decided to attempt to build a tool that would let me say, given I want X and Y, just tell me the list of raw resources I need to build. I thought this would give me a chance to play with Vue.js again and actually create something I'd use. If you aren't interested in the tech and just want to check the tool out, you can find it here:
https://cfjedimaster.github.io/nomanssky/client/index.html
Alright, now let's dig into the tech!
Getting the Data
In order to build my web app, I needed access to the data. Unfortunately, that data doesn't exist anywhere in raw form. If I wanted this information I was going to have to find a way to create it. The No Man's Sky wiki contains all of this information online in HTML format. It's maintained by the NMS community (passionate folks there) so it's always up to date. Turns out, the wiki software behind the site has an API. The API wasn't always terribly clear to me, but I was able to figure out a few things.
First, I could ask the wiki for a list of pages for a category. I found four different categories on the site that included the data I needed. Here's an example of that API call:
https://nomanssky.gamepedia.com/api.php?action=query&list=categorymembers&cmtitle=Category:Blueprints&cmlimit=500&format=json&formatversion=2
Next, I found out that you could get the "raw" data for a page by adding ?action=raw to a URL. So here is a page in regular form:
https://nomanssky.gamepedia.com/Crafting
And here is the raw form:
https://nomanssky.gamepedia.com/Crafting?action=raw
Notice the HTML is removed and it's a "data-ish" format suitable for parsing. When looking at a page for a blueprint, I saw this form was used to display crafting information:
==Crafting==
{{Craft|Carite Sheet,1;Platinum,15;Zinc,10|blueprint=yes}}
{{Repair|Carite Sheet,1;Platinum,8;Zinc,5}}
{{Dismantle|Carite Sheet,0;Platinum,7;Zinc,5}}
The first line there is for crafting, then you see repair costs and what you get if you dismantle.
My approach then was simple - get all the items that you can craft, and then suck down the "raw" form of the page and use a regex to get the data. I wrote my first script, getBuildable.js, to handle that.
/*
I get a list of things you can build, which is a combination of the Blueprints category
and Products category. Technically products have blueprints, but the wiki doesn't list things
in products under Blueprints. Not sure why.
*/
const rp = require('request-promise');
// The URL for getting blueprints
const bpURL = 'https://nomanssky.gamepedia.com/api.php?action=query&list=categorymembers&cmtitle=Category:Blueprints&cmlimit=500&format=json&formatversion=2';
// The URL for products:
const pURL = 'https://nomanssky.gamepedia.com/api.php?action=query&list=categorymembers&cmtitle=Category:Products&cmlimit=500&format=json&formatversion=2';
// The URL for consumables
const cURL = 'https://nomanssky.gamepedia.com/api.php?action=query&list=categorymembers&cmtitle=Category:Consumables&cmlimit=500&format=json&formatversion=2';
// The URL for Technology components
const tURL = 'https://nomanssky.gamepedia.com/api.php?action=query&list=categorymembers&cmtitle=Category:Technology_components&cmlimit=500&format=json&formatversion=2';
//base url for fetching page content
const rawURLBase = ' https://nomanssky.gamepedia.com/';
/*
I saw some content that is 'special' and should be ignored. Hard coded list for now.
Night Crystals needs to be checked later
*/
const IGNORE_TITLES = [
'Blueprint',
'Blueprint Trader',
'Jetpack',
'Category:Blueprint panels',
'Product',
'Alloy',
'Consumable',
'Crafting',
'Night Crystals',
'Category:Consumables',
'Nanite Tech Acquisitions'
];
Promise.all([
rp({json:true,url:bpURL}),
rp({json:true,url:pURL}),
rp({json:true,url:cURL}),
rp({json:true,url:tURL})
]).then((results) => {
let [blueprints,products,consumables,technology] = results;
let buildable = blueprints.query.categorymembers;
//we have dupes, so check first
products.query.categorymembers.forEach((prod) => {
let existing = buildable.findIndex(b => {
return b.title == prod.title;
});
if(existing === -1) buildable.push(prod);
});
consumables.query.categorymembers.forEach((con) => {
let existing = buildable.findIndex(b => {
return b.title == con.title;
});
if(existing === -1) buildable.push(con);
});
technology.query.categorymembers.forEach((tech) => {
let existing = buildable.findIndex(b => {
return b.title == tech.title;
});
if(existing === -1) buildable.push(tech);
});
//filter out specials
buildable = buildable.filter(item => {
return IGNORE_TITLES.indexOf(item.title) === -1;
});
//trim for testing
//TEMP
/*
buildable = buildable.filter(item => {
return item.title == 'Plasma Core Casing V1';
});
*/
//buildable = buildable.slice(0,90);
console.log('Total '+buildable.length + ' things to parse.');
let promises = [];
buildable.forEach(thing => {
let rawURL = `${rawURLBase}${thing.title}?action=raw`;
promises.push(rp(rawURL));
});
Promise.all(promises).then(results => {
results.forEach((result,idx) => {
let parts = getParts(result, buildable[idx].title);
buildable[idx].parts = parts;
//while we're here, lets kill ns
delete buildable[idx].ns;
});
/*
I want to signify when a part is craftable. My logic is,
if part's title is NOT in the main list, it must be a base item.
ToDo: Decide if that makes freaking sense.
*/
buildable.forEach((item, idx) => {
//Rename title to name
item.name = item.title.trim();
delete item.title;
});
//now sort by title
buildable = buildable.sort((a, b) => {
if(a.name < b.name) return -1;
if(a.name > b.name) return 1;
return 0;
});
console.log(JSON.stringify(buildable,null, '\t'));
});
});
/*
Given raw wiki text, look for:
==Crafting==
{{Craft|Name,Qty;Name2,Qty; (there is also blueprint=yes/no I may care aboyt later
*/
function getParts(s,name) {
let re = /{{Craft\|(.*?)[\||}}]+/;
let found = s.match(re);
if(!found || found.length !== 2) {
console.log(s);
throw new Error("Unable to find Craft line for "+name);
}
let productsRaw = found[1];
//productsRaw looks like: x,qty;y,qty2
let partsRaw = productsRaw.split(';');
//drop the end if it is blank
if(partsRaw[partsRaw.length-1] === '') partsRaw.pop();
let parts = [];
partsRaw.forEach((pair) => {
let [partName, partQty] = pair.split(',');
parts.push({name:partName.trim(),qty:Number(partQty)});
});
return parts;
}
For the most part, it's just "suck down a bunch of crap and parse", not anything really special, and it mostly worked on my first few tries. To run it, I did: node getBuildable.js > results.json and then simply removed the initial plain text value on top to have a valid JSON file. (Yeah, that's clunky, but I only ran it a few times.)
This gave me a JSON file of everything could be crafted and the parts necessary to do it. I thought I was done and moved on to the front end, but then I realized an issue. My goal was to generate a list of raw materials, therefore if a part of item X also has parts, I needed to recursively dig to get my list. I was going to do that on the front end, but then I thought - why do that every time? I decided to write another script that simply took the initial JSON and determined the "raw resources" necessary for every item.
Sadly, this took me like two hours. I hit a mental block with the recursion that I just struggled like hell to get past. My solution isn't even that nice, but at least I've got the "ugly" outside the web app and I can look at improving it later. Here is that script - don't laugh too loudly.
/*
So getBuildable gives us a 'pure' data list that looks like this:
{
"pageid": 4274,
"title": "Aeration Membrane Tau",
"parts": [
{
"name": "Zinc",
"qty": 50
},
{
"name": "Carbon",
"qty": 100
},
{
"name": "Microdensity Fabric",
"qty": 1,
"subcomponent": true
}
]
}
Notice that one item, Microdensity Fabric, is a subcomponent, and needs it's own stuff.
So the point of this funciton is to translate parts into a pure list of raw materials.
It will also be an array with name/qty.
*/
const fs = require('fs');
//expect the name of the JSON file
let jsonFile = process.argv[2];
let data = JSON.parse(fs.readFileSync(jsonFile, 'utf-8'));
data.forEach((item,idx) => {
//console.log('Generate raw resources for '+idx+') '+item.name +' :'+JSON.stringify(item.parts));
item.rawresources = {};
item.parts.forEach(part => {
let resources = getResources(part);
//console.log('resources for '+part.name+' was '+JSON.stringify(resources));
resources.forEach(resource => {
if(!item.rawresources[resource.name]) item.rawresources[resource.name] = 0;
item.rawresources[resource.name] += resource.qty;
});
});
//console.log('raw is '+JSON.stringify(item.rawresources,null,'\t'));
});
console.log(JSON.stringify(data,null,'\t'));
function getResources(r,arr) {
if(!arr) arr = [];
//console.log('getResource('+r.name+','+arr.toString()+')');
//Am I subcomponent?
if(!isSubcomponent(r)) {
arr.push(r);
return arr;
} else {
let subc = getSubcomponent(r);
subc.parts.forEach(part => {
let subparts = getResources(part);
subparts.forEach(subpart => {
arr.push(subpart);
});
});
return arr;
}
}
function isSubcomponent(part) {
let subc = data.findIndex(item => {
return item.name == part.name;
});
return (subc >= 0);
}
function getSubcomponent(part) {
let subc = data.findIndex(item => {
return item.name == part.name;
});
return data[subc];
}
So on to the front end. This is the second app I've built with Vue, and I still freaking love it, although I ran into some frustrations I'll explain as I go through the code. The front end is pretty simple, just three columns for layout.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>No Man's Sky Resource Tracker</title>
<link rel="stylesheet" href="app.css">
<style>
[v-cloak] {
display: none;
}
</style>
</head>
<body>
<h1>No Man's Sky Resource Tracker</h1>
<p>
The Resource Tracker lets you determine the raw materials necessary to craft items. Click on a blueprint to add it to your desired items list. Created by <a href="https://www.raymondcamden.com">Raymond Camden</a> with data from the <a href="https://nomanssky.gamepedia.com/No_Man%27s_Sky_Wiki">No Man's Sky Wiki</a>.
</p>
<div id="bpList" v-cloak class="wrapper">
<div class="one">
<h2>Blueprint List</h2>
<input type="search" placeholder="Filter" v-model="filter">
<ul class="blueprint">
<li v-for="blueprint in filteredBlueprints" @click="addToCart(blueprint.name)">{{ blueprint.name }}</li>
</ul>
</div>
<div class="two">
<h2>Desired Items</h2>
<table v-if="items.length" id="itemTable">
<thead>
<tr>
<th>Item</th>
<th>Qty</th>
<th> </th>
</tr>
</thead>
<tbody>
<tr v-for="item in items">
<td>{{item.name}}</td>
<td>{{item.qty}}</td>
<td @click="removeFromCart(item.name)" class="remove">Remove</td>
</tr>
</tbody>
</table>
</div>
<div class="three">
<h2>Resources Required</h2>
<table v-if="neededResources.length">
<thead>
<tr>
<th>Item</th>
<th>Qty</th>
</tr>
</thead>
<tbody>
<tr v-for="res in neededResources">
<td>{{res.name}}</td>
<td>{{res.qty}}</td>
</tr>
</tbody>
</table>
</div>
</div>
<script src="https://unpkg.com/vue"></script>
<script src="app.js"></script>
</body>
</html>
The Vue aspects here are pretty minimal, basically listing data. The idea is you click on a blueprint on the left side, this fills a "shopping cart" in the middle, and the right side is where your raw resource list is displayed. Now let's look at the code. (And again, this is my second Vue app, don't treat it as best practice.)
let bpApp = new Vue({
el:'#bpList',
data:{
filter:'',
blueprints:[],
items:[]
},
created:function() {
fetch('./data.json')
.then(res => res.json())
.then(res => {
this.blueprints = res;
});
},
computed:{
filteredBlueprints: function() {
let thatFilter = this.filter.toLowerCase();
return this.blueprints.filter(function(bp) {
if(thatFilter === '') return true;
if(bp.name.toLowerCase().indexOf(thatFilter) >= 0) return true;
return false;
});
},
neededResources:function() {
let needed = {};
let result = [];
/*
ok, so our shopping cart (items) has an array of items and requirements.
*/
for(let x=0;x<this.items.length;x++) {
let item = this.items[x];
//first, find it
for(let i=0;i<this.blueprints.length;i++) {
if(this.blueprints[i].name === item.name) {
for(let key in this.blueprints[i].rawresources) {
if(!needed[key]) needed[key] = 0;
needed[key] += Number(this.blueprints[i].rawresources[key]) * item.qty;
}
}
}
}
//now convert it to an array
for(let key in needed) {
result.push({name:key, qty: needed[key]});
}
result.sort(function(a,b) {
if(a.name > b.name) return 1;
if(a.name < b.name) return -1;
return 0;
});
return result;
}
},
methods:{
addToCart:function(item) {
/* why doesn't this work?
let existing = this.items.findExisting((item) => {
return item.title === item;
});
*/
let existing = -1;
for(let i=0;i<this.items.length;i++) {
if(this.items[i].name === item) existing = i;
}
if(existing === -1) {
this.items.push({name:item, qty:1});
} else {
this.items[existing].qty++;
}
this.items = this.items.sort((a,b) => {
if(a.name > b.name) return 1;
if(a.name < b.name) return -1;
return 0;
});
},
removeFromCart:function(item) {
console.log('remove '+item);
let existing = -1;
for(let i=0;i<this.items.length;i++) {
if(this.items[i].name === item) existing = i;
}
if(existing !== -1) {
//in theory it should ALWAYs match, but...
this.items.splice(existing, 1);
}
}
}
});
Ok, so let's dig into this. First, my list of blueprints is a simple array and I use the created event to handle loading it. That works awesome. But what I really freaking like is how filtering works. In my HTML, I told it to loop over filteredBlueprints, which I've defined here as a computed property. This lets me keep the original list "pure" and handle returning a list that may or may not be filtered by user input.
Note this line:
let thatFilter = this.filter.toLowerCase();
Yeah, so this is what bugged me. The Vue docs had mentioned not using arrow functions because it messes with the This scope, but then I had issues using my filter value in the callback. I got some feedback from @elpete on a gist I created to share my issues, and I think I get the reasons why, but it was still a bit awkward.
Another odd issue I ran into was in addToCart, where findExisting didn't work on the array instance. I'm still not sure why it failed and it was an easy workaround, but I'm still confused.
The final bit was the neededResources computed property which was simply a matter of looping over my "cart" and creating an object to store product requirements and totals. I then convert that object back into a simple array for rendering.
Outside of my frustrations, I am really falling hard for Vue. I love how I don't have to generate layout in my JS code. The computed properties part of this app was really freaking awesome.
As always though I'd happily take feedback on how I used Vue. If you have any suggestions, just let me know in a comment below! You can find the complete source code for both the Node parsing scripts and the front end here: https://github.com/cfjedimaster/nomanssky/.