Module: Msf::Modules::Metadata::Store

Included in:
Cache
Defined in:
lib/msf/core/modules/metadata/store.rb

Overview

Handles storage of module metadata on disk. A base metadata file is always included - this was added to ensure a much better first time user experience as generating the user based metadata file requires 100+ mb at the time of creating this module. Subsequent starts of metasploit will load from a user specific metadata file as users potentially load modules from other places.

Constant Summary collapse

BaseMetaDataFile = 
'modules_metadata_base.json'
UserMetaDataFile = 
'modules_metadata.json'
CacheMetaDataFile = 
'module_metadata_cache.json'

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.calculate_file_checksums(files, cache) ⇒ Array<Array>

Calculate checksums for all files, using the cache when possible

Parameters:

  • files (Array<String>) —

    List of file paths to check

  • cache (Hash) —

    Current cache data

Returns:

  • (Array<Array>) —

    Array of [file_path, metadata] pairs



197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
 
# File 'lib/msf/core/modules/metadata/store.rb', line 197

def self.calculate_file_checksums(files, cache)
  Parallel.map(files, in_threads: Etc.nprocessors * 2) do |file|
    # Get file metadata (size and last modified time)
    file_metadata = File.stat(file)
    cache_entry = cache[file]
    # Use cached CRC32 if mtime and size match, otherwise recalculate
    if cache_entry && cache_entry['mtime'] == file_metadata.mtime.to_i && cache_entry['size'] == file_metadata.size
      crc32 = cache_entry['crc32']
    else
      crc32 = File.open(file, 'rb') { |fd| Zlib.crc32(fd.read, 0) }
    end
    # Return file and its metadata for later aggregation
    [file, {
      'crc32' => crc32,
      'mtime' => file_metadata.mtime.to_i,
      'size' => file_metadata.size
    }]
  end
end

.calculate_overall_checksum(file_crc32s) ⇒ Integer

Calculate the overall checksum from individual file checksums

Parameters:

  • file_crc32s (Array<Integer>) —

    Array of individual file CRC32 values

Returns:

  • (Integer) —

    The overall CRC32 as an integer



181
182
183
 
# File 'lib/msf/core/modules/metadata/store.rb', line 181

def self.calculate_overall_checksum(file_crc32s)
  Zlib.crc32(file_crc32s.join(','), 0)
end

.checksums_match?(current_checksum, cached_checksum) ⇒ Boolean

Compare the current checksum with the cached checksum

Parameters:

  • current_checksum (String) —

    The calculated checksum for the current state

  • cached_checksum (String) —

    The checksum retrieved from cache

Returns:

  • (Boolean) —

    True if checksums match, false otherwise



174
175
176
 
# File 'lib/msf/core/modules/metadata/store.rb', line 174

def self.checksums_match?(current_checksum, cached_checksum)
  current_checksum == cached_checksum
end

.collect_files_to_check ⇒ Array<String>

Collect all files that need to be checked for checksums.

Returns:

  • (Array<String>) —

    List of file paths



187
188
189
190
191
 
# File 'lib/msf/core/modules/metadata/store.rb', line 187

def self.collect_files_to_check
  modules_dir = File.join(Msf::Config.install_root, 'modules', '**', '*')
  local_modules_dir = File.join(Msf::Config.user_module_directory, '**', '*')
  Dir.glob([modules_dir, local_modules_dir]).select { |f| File.file?(f) }.sort
end

.get_cache_path ⇒ String

Get the path to the cache file

Returns:

  • (String) —

    Path to the cache file



219
220
221
 
# File 'lib/msf/core/modules/metadata/store.rb', line 219

def self.get_cache_path
  File.join(Msf::Config.config_directory, "store", CacheMetaDataFile)
end

.get_cached_checksum ⇒ Integer?

Get the cached checksum value from the combined cache file

Returns:

  • (Integer, nil) —

    The cached checksum value or nil if no cache exists



258
259
260
261
262
 
# File 'lib/msf/core/modules/metadata/store.rb', line 258

def self.get_cached_checksum
  cache_path = get_cache_path
  cache_data = load_combined_cache(cache_path)
  cache_data['checksum']
end

.get_current_checksum ⇒ Integer

Calculate the current checksum for all module and library files This calculates checksums for each file and generates an overall checksum from the individual file checksums. Does NOT update the cached checksum.

Returns:

  • (Integer) —

    The current overall checksum



153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/msf/core/modules/metadata/store.rb', line 153

def self.get_current_checksum
  files = collect_files_to_check
  cache_file = get_cache_path
  cache_data = load_combined_cache(cache_file)

  files_lookup = {}
  cache_data['files'].each { |entry| files_lookup[entry['path']] = entry }

  file_crc32s_with_metadata = calculate_file_checksums(files, files_lookup)

  file_crc32s = file_crc32s_with_metadata.map { |_, meta| meta['crc32'] }.sort

  overall_checksum = calculate_overall_checksum(file_crc32s)

  overall_checksum
end

.load_combined_cache(cache_file) ⇒ Hash

Load the combined cache from disk (contains both files and checksum)

Parameters:

  • cache_file (String) —

    Path to the cache file

Returns:

  • (Hash) —

    The loaded cache with 'files' and 'checksum' keys, or empty structure if file doesn't exist



226
227
228
229
230
231
232
233
234
235
236
237
 
# File 'lib/msf/core/modules/metadata/store.rb', line 226

def self.load_combined_cache(cache_file)
  if File.exist?(cache_file)
    cache_content = JSON.parse(File.read(cache_file))
    # Ensure the cache has the expected structure
    {
      'files' => cache_content['files'] || [],
      'checksum' => cache_content['checksum']
    }
  else
    { 'files' => [], 'checksum' => nil }
  end
end

.save_combined_cache(cache_file, files_cache, overall_checksum) ⇒ void

This method returns an undefined value.

Save the combined cache to disk (files and checksum in one file)

Parameters:

  • cache_file (String) —

    Path to the cache file

  • files_cache (Hash) —

    The per-file cache data

  • overall_checksum (Integer) —

    The overall checksum



244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/msf/core/modules/metadata/store.rb', line 244

def self.save_combined_cache(cache_file, files_cache, overall_checksum)
  # Ensure the directory for the cache file exists before writing
  FileUtils.mkdir_p(File.dirname(cache_file))

  cache_content = {
    'checksum' => overall_checksum,
    'files' => files_cache
  }

  File.write(cache_file, JSON.pretty_generate(cache_content))
end

.update_cache_checksum(current_checksum) ⇒ void

This method returns an undefined value.

Update the cache with the current checksum and file data

Parameters:

  • current_checksum (Integer) —

    The current checksum to store in the cache



267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
 
# File 'lib/msf/core/modules/metadata/store.rb', line 267

def self.update_cache_checksum(current_checksum)
  # Recalculate file checksums and update both overall checksum and file cache
  files = collect_files_to_check
  cache_file = get_cache_path
  cache_data = load_combined_cache(cache_file)

  files_lookup = {}
  cache_data['files'].each { |entry| files_lookup[entry['path']] = entry }

  file_crc32s_with_metadata = calculate_file_checksums(files, files_lookup)

  updated_files_cache = file_crc32s_with_metadata.map do |file_path, metadata|
    metadata.merge('path' => file_path)
  end

  updated_files_cache.sort_by! { |entry| entry['path'] }

  # Save both the updated file cache and the new overall checksum
  save_combined_cache(cache_file, updated_files_cache, current_checksum)
end

.valid_checksum? ⇒ Boolean

This method checks if the current module and library files match the cached checksum. It uses a per-file CRC32 cache to avoid recalculating checksums for files that haven't changed. If no cache exists, it will create one in the user's directory.

Returns:

  • (Boolean) —

    True if the current checksum matches the cached one



135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/msf/core/modules/metadata/store.rb', line 135

def self.valid_checksum?
  current_checksum = get_current_checksum
  cached_sha = get_cached_checksum

  # If no cached checksum exists, create the cache file with current checksum
  if cached_sha.nil?
    update_cache_checksum(current_checksum)
    return false
  end

  checksums_match?(current_checksum, cached_sha)
end

Instance Method Details

#init_store ⇒ Object

Initializes from user store (under ~/store/.msf4) if it exists. else base file (under $INSTALL_ROOT/db) is copied and loaded.



24
25
26
 
# File 'lib/msf/core/modules/metadata/store.rb', line 24

def init_store
  load_metadata
end

#initialize ⇒ Object 



13
14
15
 
# File 'lib/msf/core/modules/metadata/store.rb', line 13

def initialize
  @update_mutex = Mutex.new
end